Xel: A cloud-agnostic data platform for the design-driven building of high-availability data science services

J. Armando Barrón-Lugo, J.L.Gonzalez-Compean, Ivan Lopez-Arevalo, Jesus Carretero and Jose L. Martinez-Rodriguez

Abstract

This paper presents Xel, a cloud-agnostic data platform for the design-driven building of high-availability data science services as a support tool for data-driven decision-making. We designed and implemented Xel based on four main components: (a) a high level and driven-design framework for end-users to select analytic and machine learning tools from a service mesh and coupling them into the form of processing pipelines; (b) a new recursive ETL processing model to automatically convert the pipeline designs into infrastructure-agnostic software structures, which are deployed on multiple infrastructures; (c) an orchestration model for transparently managing the data delivery throughout each stage of the processing pipelines used in data science systems; and (d) a data decentralized model to transparently mask service unavailability such as cloud outages and unavailability of either applications or data. Real users created, by means of Xel, data science services such as deep learning analysis of scientific publications, clustering of movie reviews, and a cancer exploratory study. These services were evaluated as case studies that revealed the efficacy of this platform design for enabling end-users to create multiple types of data science pipelines without programming nor making configurations, and automatically masking unavailability of cloud resources and data. This platform is currently used to create a national cancer observatory and big data systems for fusing suicide, mental health, drug consumption and macroeconomic datasets to find spatiotemporal patterns.

https://doi.org/10.1016/j.future.2023.03.019