A Comprehensive Multi-Vocal Empirical Study of ML Cloud Service Misuses

• URL: http://arxiv.org/abs/2503.09815v2
• Date: Mon, 20 Oct 2025 22:32:49 GMT
• Title: A Comprehensive Multi-Vocal Empirical Study of ML Cloud Service Misuses
• Abstract summary: We propose a catalog of 20 ML cloud service misuses, most of which have not been studied in prior research.<n>Our results show that ML service misuses are common in both open-source projects and industry.<n>This emphasizes the importance of ongoing education in best practices for ML services.
• Score: 0.8156494881838944
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine Learning (ML) models are widely used across various domains, including medical diagnostics and autonomous driving. To support this growth, cloud providers offer ML services to ease the integration of ML components in software systems. The evolving business requirements and the popularity of ML services have led practitioners of all skill levels to implement, and maintain ML service-based systems. However, they may not always adhere to optimal design and usage practices for ML cloud services, resulting in common misuse which could significantly degrade the quality of ML service-based systems and adversely affect their maintenance and evolution. Though much research has been conducted on ML service misuse, a consistent terminology and specification for these misuses remain absent. We therefore conduct in this paper a comprehensive, multi-vocal empirical study exploring the prevalence of ML cloud service misuses in practice. We propose a catalog of 20 ML cloud service misuses, most of which have not been studied in prior research. To achieve this, we conducted a) a systematic literature review of studies on ML misuses, b) a gray literature review of the official documentation provided by major cloud providers, c) an empirical analysis of a curated set of 377 ML service-based systems on GitHub, and d) a survey with 50 ML practitioners. Our results show that ML service misuses are common in both open-source projects and industry, often stemming from a lack of understanding of service capabilities, and insufficient documentation. This emphasizes the importance of ongoing education in best practices for ML services, which is the focus of this paper, while also highlighting the need for tools to automatically detect and refactor ML misuses.

Related papers

• MLmisFinder: A Specification and Detection Approach of Machine Learning Service Misuses [1.2955713934068565]
  Machine learning (ML) cloud services enable the integration of ML components into software systems without building models from scratch.<n>Rapid adoption of ML services has led to widespread misuses, compromising quality, maintainability, and evolution of ML service-based systems.<n>We propose MLmisFinder, an automatic approach to detect ML service misuses in software systems.
  arXiv  Detail & Related papers  (2026-03-18T03:48:20Z)
• Empowering Multimodal LLMs with External Tools: A Comprehensive Survey [61.66069828956139]
  Multimodal Large Language Models (MLLMs) have achieved great success in various multimodal tasks, pointing toward a promising pathway to artificial general intelligence.<n>Lack of multimodal data, poor performance on many complex downstream tasks, and inadequate evaluation protocols hinder the reliability and broader applicability of MLLMs.<n>Inspired by the human ability to leverage external tools for enhanced reasoning and problem-solving, augmenting MLLMs with external tools offers a promising strategy to overcome these challenges.
  arXiv  Detail & Related papers  (2025-08-14T07:25:45Z)
• MLZero: A Multi-Agent System for End-to-end Machine Learning Automation [48.716299953336346]
  We introduce MLZero, a novel multi-agent framework powered by Large Language Models (LLMs)<n>A cognitive perception module is first employed, transforming raw multimodal inputs into perceptual context.<n> MLZero demonstrates superior performance on MLE-Bench Lite, outperforming all competitors in both success rate and solution quality.
  arXiv  Detail & Related papers  (2025-05-20T05:20:53Z)
• Benchmarking Large and Small MLLMs [71.78055760441256]
  Large multimodal language models (MLLMs) have achieved remarkable advancements in understanding and generating multimodal content. However, their deployment faces significant challenges, including slow inference, high computational cost, and impracticality for on-device applications. Small MLLMs, exemplified by the LLava-series models and Phi-3-Vision, offer promising alternatives with faster inference, reduced deployment costs, and the ability to handle domain-specific scenarios.
  arXiv  Detail & Related papers  (2025-01-04T07:44:49Z)
• Identifying Concerns When Specifying Machine Learning-Enabled Systems: A Perspective-Based Approach [1.2184324428571227]
  PerSpecML is a perspective-based approach for specifying ML-enabled systems. It helps practitioners identify which attributes, including ML and non-ML components, are important to contribute to the overall system's quality.
  arXiv  Detail & Related papers  (2023-09-14T18:31:16Z)
• Vulnerability of Machine Learning Approaches Applied in IoT-based Smart Grid: A Review [51.31851488650698]
  Machine learning (ML) sees an increasing prevalence of being used in the internet-of-things (IoT)-based smart grid. adversarial distortion injected into the power signal will greatly affect the system's normal control and operation. It is imperative to conduct vulnerability assessment for MLsgAPPs applied in the context of safety-critical power systems.
  arXiv  Detail & Related papers  (2023-08-30T03:29:26Z)
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models [73.86954509967416]
  Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks. This paper presents the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks.
  arXiv  Detail & Related papers  (2023-06-23T09:22:36Z)
• Machine Learning for QoS Prediction in Vehicular Communication: Challenges and Solution Approaches [46.52224306624461]
  We consider maximum throughput prediction enhancing, for example, streaming or high-definition mapping applications. We highlight how confidence can be built on machine learning technologies by better understanding the underlying characteristics of the collected data. We use explainable AI to show that machine learning can learn underlying principles of wireless networks without being explicitly programmed.
  arXiv  Detail & Related papers  (2023-02-23T12:29:20Z)
• Understanding the Complexity and Its Impact on Testing in ML-Enabled Systems [8.630445165405606]
  We study Rasa 3.0, an industrial dialogue system that has been widely adopted by various companies around the world. Our goal is to characterize the complexity of such a largescale ML-enabled system and to understand the impact of the complexity on testing. Our study reveals practical implications for software engineering for ML-enabled systems.
  arXiv  Detail & Related papers  (2023-01-10T08:13:24Z)
• Towards Perspective-Based Specification of Machine Learning-Enabled Systems [1.3406258114080236]
  This paper describes our work towards a perspective-based approach for specifying ML-enabled systems. The approach involves analyzing a set of 45 ML concerns grouped into five perspectives: objectives, user experience, infrastructure, model, and data. The main contribution of this paper is to provide two new artifacts that can be used to help specifying ML-enabled systems.
  arXiv  Detail & Related papers  (2022-06-20T13:09:23Z)
• Panoramic Learning with A Standardized Machine Learning Formalism [116.34627789412102]
  This paper presents a standardized equation of the learning objective, that offers a unifying understanding of diverse ML algorithms. It also provides guidance for mechanic design of new ML solutions, and serves as a promising vehicle towards panoramic learning with all experiences.
  arXiv  Detail & Related papers  (2021-08-17T17:44:38Z)
• Quality Assurance Challenges for Machine Learning Software Applications During Software Development Life Cycle Phases [1.4213973379473654]
  The paper conducts an in-depth review of literature on the quality assurance of Machine Learning (ML) models. We develop a taxonomy of MLSA quality assurance issues by mapping the various ML adoption challenges across different phases of software development life cycles (SDLC) This mapping can help prioritize quality assurance efforts of MLSAs where the adoption of ML models can be considered crucial.
  arXiv  Detail & Related papers  (2021-05-03T22:29:23Z)
• Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems [1.4695979686066065]
  Development and deployment of machine learning systems remains a challenge. In this paper, we report our findings and their implications for improving end-to-end ML-enabled system development.
  arXiv  Detail & Related papers  (2021-03-25T19:40:29Z)
• Technology Readiness Levels for Machine Learning Systems [107.56979560568232]
  Development and deployment of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. We have developed a proven systems engineering approach for machine learning development and deployment. Our "Machine Learning Technology Readiness Levels" framework defines a principled process to ensure robust, reliable, and responsible systems.
  arXiv  Detail & Related papers  (2021-01-11T15:54:48Z)
• Technology Readiness Levels for AI & ML [79.22051549519989]
  Development of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. Engineering systems follow well-defined processes and testing standards to streamline development for high-quality, reliable results. We propose a proven systems engineering approach for machine learning development and deployment.
  arXiv  Detail & Related papers  (2020-06-21T17:14:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.