Towards Perspective-Based Specification of Machine Learning-Enabled Systems

• URL: http://arxiv.org/abs/2206.09760v1
• Date: Mon, 20 Jun 2022 13:09:23 GMT
• Title: Towards Perspective-Based Specification of Machine Learning-Enabled Systems
• Authors: Hugo Villamizar, Marcos Kalinowski, and Helio Lopes
• Abstract summary: This paper describes our work towards a perspective-based approach for specifying ML-enabled systems. The approach involves analyzing a set of 45 ML concerns grouped into five perspectives: objectives, user experience, infrastructure, model, and data. The main contribution of this paper is to provide two new artifacts that can be used to help specifying ML-enabled systems.
• Score: 1.3406258114080236
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning (ML) teams often work on a project just to realize the performance of the model is not good enough. Indeed, the success of ML-enabled systems involves aligning data with business problems, translating them into ML tasks, experimenting with algorithms, evaluating models, capturing data from users, among others. Literature has shown that ML-enabled systems are rarely built based on precise specifications for such concerns, leading ML teams to become misaligned due to incorrect assumptions, which may affect the quality of such systems and overall project success. In order to help addressing this issue, this paper describes our work towards a perspective-based approach for specifying ML-enabled systems. The approach involves analyzing a set of 45 ML concerns grouped into five perspectives: objectives, user experience, infrastructure, model, and data. The main contribution of this paper is to provide two new artifacts that can be used to help specifying ML-enabled systems: (i) the perspective-based ML task and concern diagram and (ii) the perspective-based ML specification template.

Related papers

• Verbalized Machine Learning: Revisiting Machine Learning with Language Models [63.10391314749408]
  We introduce the framework of verbalized machine learning (VML) VML constrains the parameter space to be human-interpretable natural language. We empirically verify the effectiveness of VML, and hope that VML can serve as a stepping stone to stronger interpretability.
  arXiv  Detail & Related papers  (2024-06-06T17:59:56Z)
• Understanding Information Storage and Transfer in Multi-modal Large Language Models [51.20840103605018]
  We study how Multi-modal Large Language Models process information in a factual visual question answering task. Key findings show that these MLLMs rely on self-attention blocks in much earlier layers for information storage. We introduce MultEdit, a model-editing algorithm that can correct errors and insert new long-tailed information into MLLMs.
  arXiv  Detail & Related papers  (2024-06-06T16:35:36Z)
• Characterization of Large Language Model Development in the Datacenter [55.9909258342639]
  Large Language Models (LLMs) have presented impressive performance across several transformative tasks. However, it is non-trivial to efficiently utilize large-scale cluster resources to develop LLMs. We present an in-depth characterization study of a six-month LLM development workload trace collected from our GPU datacenter Acme.
  arXiv  Detail & Related papers  (2024-03-12T13:31:14Z)
• LLM Inference Unveiled: Survey and Roofline Model Insights [62.92811060490876]
  Large Language Model (LLM) inference is rapidly evolving, presenting a unique blend of opportunities and challenges. Our survey stands out from traditional literature reviews by not only summarizing the current state of research but also by introducing a framework based on roofline model. This framework identifies the bottlenecks when deploying LLMs on hardware devices and provides a clear understanding of practical problems.
  arXiv  Detail & Related papers  (2024-02-26T07:33:05Z)
• ML-Bench: Evaluating Large Language Models and Agents for Machine Learning Tasks on Repository-Level Code [76.84199699772903]
  ML-Bench is a benchmark rooted in real-world programming applications that leverage existing code repositories to perform tasks. To evaluate both Large Language Models (LLMs) and AI agents, two setups are employed: ML-LLM-Bench for assessing LLMs' text-to-code conversion within a predefined deployment environment, and ML-Agent-Bench for testing autonomous agents in an end-to-end task execution within a Linux sandbox environment.
  arXiv  Detail & Related papers  (2023-11-16T12:03:21Z)
• Identifying Concerns When Specifying Machine Learning-Enabled Systems: A Perspective-Based Approach [1.2184324428571227]
  PerSpecML is a perspective-based approach for specifying ML-enabled systems. It helps practitioners identify which attributes, including ML and non-ML components, are important to contribute to the overall system's quality.
  arXiv  Detail & Related papers  (2023-09-14T18:31:16Z)
• Understanding the Complexity and Its Impact on Testing in ML-Enabled Systems [8.630445165405606]
  We study Rasa 3.0, an industrial dialogue system that has been widely adopted by various companies around the world. Our goal is to characterize the complexity of such a largescale ML-enabled system and to understand the impact of the complexity on testing. Our study reveals practical implications for software engineering for ML-enabled systems.
  arXiv  Detail & Related papers  (2023-01-10T08:13:24Z)
• Prescriptive and Descriptive Approaches to Machine-Learning Transparency [5.040810032102723]
  We propose a preliminary approach, called Method Cards, which aims to increase the transparency and prescriptive documentation of machine-learning systems. We showcase our proposal with an example in small object detection, and demonstrate how Method Cards can communicate key considerations for model developers.
  arXiv  Detail & Related papers  (2022-04-27T15:26:50Z)
• Characterizing and Detecting Mismatch in Machine-Learning-Enabled Systems [1.4695979686066065]
  Development and deployment of machine learning systems remains a challenge. In this paper, we report our findings and their implications for improving end-to-end ML-enabled system development.
  arXiv  Detail & Related papers  (2021-03-25T19:40:29Z)
• A Survey on Large-scale Machine Learning [67.6997613600942]
  Machine learning can provide deep insights into data, allowing machines to make high-quality predictions. Most sophisticated machine learning approaches suffer from huge time costs when operating on large-scale data. Large-scale Machine Learning aims to learn patterns from big data with comparable performance efficiently.
  arXiv  Detail & Related papers  (2020-08-10T06:07:52Z)
• Insights into Performance Fitness and Error Metrics for Machine Learning [1.827510863075184]
  Machine learning (ML) is the field of training machines to achieve high level of cognition and perform human-like analysis. This paper examines a number of the most commonly-used performance fitness and error metrics for regression and classification algorithms.
  arXiv  Detail & Related papers  (2020-05-17T22:59:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.