Understanding the Complexity and Its Impact on Testing in ML-Enabled Systems

• URL: http://arxiv.org/abs/2301.03837v1
• Date: Tue, 10 Jan 2023 08:13:24 GMT
• Title: Understanding the Complexity and Its Impact on Testing in ML-Enabled Systems
• Authors: Junming Cao, Bihuan Chen, Longjie Hu, Jie Gao, Kaifeng Huang, Xin Peng
• Abstract summary: We study Rasa 3.0, an industrial dialogue system that has been widely adopted by various companies around the world. Our goal is to characterize the complexity of such a largescale ML-enabled system and to understand the impact of the complexity on testing. Our study reveals practical implications for software engineering for ML-enabled systems.
• Score: 8.630445165405606
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning (ML) enabled systems are emerging with recent breakthroughs in ML. A model-centric view is widely taken by the literature to focus only on the analysis of ML models. However, only a small body of work takes a system view that looks at how ML components work with the system and how they affect software engineering for MLenabled systems. In this paper, we adopt this system view, and conduct a case study on Rasa 3.0, an industrial dialogue system that has been widely adopted by various companies around the world. Our goal is to characterize the complexity of such a largescale ML-enabled system and to understand the impact of the complexity on testing. Our study reveals practical implications for software engineering for ML-enabled systems.

Related papers

• A Large-Scale Study of Model Integration in ML-Enabled Software Systems [4.776073133338119]
  Machine learning (ML) and its embedding in systems has drastically changed the engineering of software-intensive systems. Traditionally, software engineering focuses on manually created artifacts such as source code and the process of creating them. We present the first large-scale study of real ML-enabled software systems, covering over 2,928 open source systems on GitHub.
  arXiv  Detail & Related papers  (2024-08-12T15:28:40Z)
• A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks [74.52259252807191]
  Multimodal Large Language Models (MLLMs) address the complexities of real-world applications far beyond the capabilities of single-modality systems. This paper systematically sorts out the applications of MLLM in multimodal tasks such as natural language, vision, and audio.
  arXiv  Detail & Related papers  (2024-08-02T15:14:53Z)
• Efficient Multimodal Large Language Models: A Survey [60.7614299984182]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable performance in tasks such as visual question answering, visual understanding and reasoning. The extensive model size and high training and inference costs have hindered the widespread application of MLLMs in academia and industry. This survey provides a comprehensive and systematic review of the current state of efficient MLLMs.
  arXiv  Detail & Related papers  (2024-05-17T12:37:10Z)
• Vulnerability of Machine Learning Approaches Applied in IoT-based Smart Grid: A Review [51.31851488650698]
  Machine learning (ML) sees an increasing prevalence of being used in the internet-of-things (IoT)-based smart grid. adversarial distortion injected into the power signal will greatly affect the system's normal control and operation. It is imperative to conduct vulnerability assessment for MLsgAPPs applied in the context of safety-critical power systems.
  arXiv  Detail & Related papers  (2023-08-30T03:29:26Z)
• An Exploratory Study of V-Model in Building ML-Enabled Software: A Systems Engineering Perspective [0.7252027234425334]
  Machine learning (ML) components are being added to more and more critical and impactful software systems. This research investigates the use of V-Model in addressing the interdisciplinary collaboration challenges when building ML-enabled systems.
  arXiv  Detail & Related papers  (2023-08-10T06:53:32Z)
• Towards Perspective-Based Specification of Machine Learning-Enabled Systems [1.3406258114080236]
  This paper describes our work towards a perspective-based approach for specifying ML-enabled systems. The approach involves analyzing a set of 45 ML concerns grouped into five perspectives: objectives, user experience, infrastructure, model, and data. The main contribution of this paper is to provide two new artifacts that can be used to help specifying ML-enabled systems.
  arXiv  Detail & Related papers  (2022-06-20T13:09:23Z)
• Tiny Robot Learning: Challenges and Directions for Machine Learning in Resource-Constrained Robots [57.27442333662654]
  Machine learning (ML) has become a pervasive tool across computing systems. Tiny robot learning is the deployment of ML on resource-constrained low-cost autonomous robots. Tiny robot learning is subject to challenges from size, weight, area, and power (SWAP) constraints. This paper gives a brief survey of the tiny robot learning space, elaborates on key challenges, and proposes promising opportunities for future work in ML system design.
  arXiv  Detail & Related papers  (2022-05-11T19:36:15Z)
• Declarative Machine Learning Systems [7.5717114708721045]
  Machine learning (ML) has moved from a academic endeavor to a pervasive technology adopted in almost every aspect of computing. Recent successes in applying ML in natural sciences revealed that ML can be used to tackle some of the hardest real-world problems humanity faces today. We believe the next wave of ML systems will allow a larger amount of people, potentially without coding skills, to perform the same tasks.
  arXiv  Detail & Related papers  (2021-07-16T23:57:57Z)
• Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems [1.4695979686066065]
  Development and deployment of machine learning systems remains a challenge. In this paper, we report our findings and their implications for improving end-to-end ML-enabled system development.
  arXiv  Detail & Related papers  (2021-03-25T19:40:29Z)
• White Paper Machine Learning in Certified Systems [70.24215483154184]
  DEEL Project set-up the ML Certification 3 Workgroup (WG) set-up by the Institut de Recherche Technologique Saint Exup'ery de Toulouse (IRT)
  arXiv  Detail & Related papers  (2021-03-18T21:14:30Z)
• Technology Readiness Levels for Machine Learning Systems [107.56979560568232]
  Development and deployment of machine learning systems can be executed easily with modern tools, but the process is typically rushed and means-to-an-end. We have developed a proven systems engineering approach for machine learning development and deployment. Our "Machine Learning Technology Readiness Levels" framework defines a principled process to ensure robust, reliable, and responsible systems.
  arXiv  Detail & Related papers  (2021-01-11T15:54:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.