Accelerated Cloud for Artificial Intelligence (ACAI)

• URL: http://arxiv.org/abs/2401.16791v1
• Date: Tue, 30 Jan 2024 07:09:48 GMT
• Title: Accelerated Cloud for Artificial Intelligence (ACAI)
• Authors: Dachi Chen, Weitian Ding, Chen Liang, Chang Xu, Junwei Zhang, Majd Sakr
• Abstract summary: We propose an end-to-end cloud-based machine learning platform, Accelerated Cloud for AI (ACAI) ACAI enables cloud-based storage of indexed, labeled, and searchable data, as well as automatic resource provisioning, job scheduling, and experiment tracking. We show that our auto-provisioner produces a 1.7x speed-up and 39% cost reduction, and our system reduces experiment time for ML scientists by 20% on typical ML use cases.
• Score: 24.40451195277244
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training an effective Machine learning (ML) model is an iterative process that requires effort in multiple dimensions. Vertically, a single pipeline typically includes an initial ETL (Extract, Transform, Load) of raw datasets, a model training stage, and an evaluation stage where the practitioners obtain statistics of the model performance. Horizontally, many such pipelines may be required to find the best model within a search space of model configurations. Many practitioners resort to maintaining logs manually and writing simple glue code to automate the workflow. However, carrying out this process on the cloud is not a trivial task in terms of resource provisioning, data management, and bookkeeping of job histories to make sure the results are reproducible. We propose an end-to-end cloud-based machine learning platform, Accelerated Cloud for AI (ACAI), to help improve the productivity of ML practitioners. ACAI achieves this goal by enabling cloud-based storage of indexed, labeled, and searchable data, as well as automatic resource provisioning, job scheduling, and experiment tracking. Specifically, ACAI provides practitioners (1) a data lake for storing versioned datasets and their corresponding metadata, and (2) an execution engine for executing ML jobs on the cloud with automatic resource provisioning (auto-provision), logging and provenance tracking. To evaluate ACAI, we test the efficacy of our auto-provisioner on the MNIST handwritten digit classification task, and we study the usability of our system using experiments and interviews. We show that our auto-provisioner produces a 1.7x speed-up and 39% cost reduction, and our system reduces experiment time for ML scientists by 20% on typical ML use cases.

Related papers

• Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration [90.41908331897639]
  Large language models (LLMs) have significantly benefited from training on diverse, high-quality task-specific data. We present a novel approach, ReverseGen, designed to automatically generate effective training samples.
  arXiv  Detail & Related papers  (2024-10-22T06:43:28Z)
• AutoML-Agent: A Multi-Agent LLM Framework for Full-Pipeline AutoML [56.565200973244146]
  Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline. Recent works have started exploiting large language models (LLM) to lessen such burden. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML.
  arXiv  Detail & Related papers  (2024-10-03T20:01:09Z)
• Automated Machine Learning in Insurance [0.0]
  This paper introduces an AutoML workflow that allows users without domain knowledge or prior experience to achieve robust and effortless ML deployment by writing only a few lines of code. This proposed AutoML is specifically tailored for the insurance application, with features like the balancing step in data preprocessing, ensemble pipelines, and customized loss functions.
  arXiv  Detail & Related papers  (2024-08-26T14:55:40Z)
• Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows? [73.81908518992161]
  We introduce Spider2-V, the first multimodal agent benchmark focusing on professional data science and engineering. Spider2-V features real-world tasks in authentic computer environments and incorporating 20 enterprise-level professional applications. These tasks evaluate the ability of a multimodal agent to perform data-related tasks by writing code and managing the GUI in enterprise data software systems.
  arXiv  Detail & Related papers  (2024-07-15T17:54:37Z)
• DiffPrep: Differentiable Data Preprocessing Pipeline Search for Learning over Tabular Data [12.416345241511781]
  We propose DiffPrep to automatically and efficiently search for a data preprocessing pipeline for a given dataset. Our experiments show that DiffPrep achieves the best test accuracy on 15 out of the 18 real-world datasets evaluated.
  arXiv  Detail & Related papers  (2023-08-20T23:40:26Z)
• Learning Representations on Logs for AIOps [6.47086647390439]
  Large Language Models (LLMs) are trained using self-supervision on a vast amount of unlabeled data. This paper introduces a LLM for log data which is trained on public and proprietary log data. Our proposed LLM, trained on public and proprietary log data, offers superior performance on multiple downstream tasks.
  arXiv  Detail & Related papers  (2023-08-18T20:34:46Z)
• OmniForce: On Human-Centered, Large Model Empowered and Cloud-Edge Collaborative AutoML System [85.8338446357469]
  We introduce OmniForce, a human-centered AutoML system that yields both human-assisted ML and ML-assisted human techniques. We show how OmniForce can put an AutoML system into practice and build adaptive AI in open-environment scenarios.
  arXiv  Detail & Related papers  (2023-03-01T13:35:22Z)
• SOLIS -- The MLOps journey from data acquisition to actionable insights [62.997667081978825]
  In this paper we present a unified deployment pipeline and freedom-to-operate approach that supports all requirements while using basic cross-platform tensor framework and script language engines. This approach however does not supply the needed procedures and pipelines for the actual deployment of machine learning capabilities in real production grade systems.
  arXiv  Detail & Related papers  (2021-12-22T14:45:37Z)
• Automated Machine Learning Techniques for Data Streams [91.3755431537592]
  This paper surveys the state-of-the-art open-source AutoML tools, applies them to data collected from streams, and measures how their performance changes over time. The results show that off-the-shelf AutoML tools can provide satisfactory results but in the presence of concept drift, detection or adaptation techniques have to be applied to maintain the predictive accuracy over time.
  arXiv  Detail & Related papers  (2021-06-14T11:42:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.