A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners

• URL: http://arxiv.org/abs/2304.00078v1
• Date: Fri, 31 Mar 2023 19:00:59 GMT
• Title: A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners
• Abstract summary: Incorporating machine learning (ML) components into software products raises new software-engineering challenges. We collect 50 relevant papers that together interacted with over 4758 practitioners using guidelines for systematic literature reviews. We then collected, grouped, and organized the over 500 mentions of challenges within those papers.
• Score: 13.565678193855609
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Incorporating machine learning (ML) components into software products raises new software-engineering challenges and exacerbates existing challenges. Many researchers have invested significant effort in understanding the challenges of industry practitioners working on building products with ML components, through interviews and surveys with practitioners. With the intention to aggregate and present their collective findings, we conduct a meta-summary study: We collect 50 relevant papers that together interacted with over 4758 practitioners using guidelines for systematic literature reviews. We then collected, grouped, and organized the over 500 mentions of challenges within those papers. We highlight the most commonly reported challenges and hope this meta-summary will be a useful resource for the research community to prioritize research and education in this field.

Related papers

• Socio-Technical Anti-Patterns in Building ML-Enabled Software: Insights from Leaders on the Forefront [4.692400531340393]
  We focus on socio-technical challenges of productionizing machine learning models centered around and within teams.<n>By analyzing talks from practitioners for practitioners of a community with over 11,000 members in their Slack workspace, we found 17 anti-patterns.<n>We list recommendations to overcome these problems, ranging from technical solutions over guidelines to organizational restructuring.
  arXiv  Detail & Related papers  (2026-07-03T12:38:30Z)
• Deep Research: A Systematic Survey [118.82795024422722]
  Deep Research (DR) aims to combine the reasoning capabilities of large language models with external tools, such as search engines.<n>This survey presents a comprehensive and systematic overview of deep research systems.
  arXiv  Detail & Related papers  (2025-11-24T15:28:28Z)
• Large Language Models for Unit Testing: A Systematic Literature Review [13.465045964128196]
  This paper presents the first systematic literature review on the application of Large Language Models in unit testing until March 2025.<n>We first categorize existing unit testing tasks that benefit from LLMs, e.g., test generation and oracle generation.<n>We then discuss several critical aspects of integrating LLMs into unit testing research, including model usage, adaptation strategies, and hybrid approaches.
  arXiv  Detail & Related papers  (2025-06-18T08:11:10Z)
• Can LLMs Generate Tabular Summaries of Science Papers? Rethinking the Evaluation Protocol [83.90769864167301]
  Literature review tables are essential for summarizing and comparing collections of scientific papers. We explore the task of generating tables that best fulfill a user's informational needs given a collection of scientific papers. Our contributions focus on three key challenges encountered in real-world use: (i) User prompts are often under-specified; (ii) Retrieved candidate papers frequently contain irrelevant content; and (iii) Task evaluation should move beyond shallow text similarity techniques.
  arXiv  Detail & Related papers  (2025-04-14T14:52:28Z)
• Practitioners' Discussions on Building LLM-based Applications for Production [6.544757635738911]
  We collected 189 videos from 2022 to 2024 from practitioners actively developing large language models (LLMs) We analyzed the transcripts using BERTopic, then manually sorted and merged the generated topics into themes, leading to a total of 20 topics in 8 themes. The most prevalent topics fall within the theme Design & Architecture, with a strong focus on retrieval-augmented generation (RAG) systems.
  arXiv  Detail & Related papers  (2024-11-13T12:44:41Z)
• SUPER: Evaluating Agents on Setting Up and Executing Tasks from Research Repositories [55.161075901665946]
  Super aims to capture the realistic challenges faced by researchers working with Machine Learning (ML) and Natural Language Processing (NLP) research repositories. Our benchmark comprises three distinct problem sets: 45 end-to-end problems with annotated expert solutions, 152 sub problems derived from the expert set that focus on specific challenges, and 602 automatically generated problems for larger-scale development. We show that state-of-the-art approaches struggle to solve these problems with the best model (GPT-4o) solving only 16.3% of the end-to-end set, and 46.1% of the scenarios.
  arXiv  Detail & Related papers  (2024-09-11T17:37:48Z)
• Retrieval-Enhanced Machine Learning: Synthesis and Opportunities [60.34182805429511]
  Retrieval-enhancement can be extended to a broader spectrum of machine learning (ML) This work introduces a formal framework of this paradigm, Retrieval-Enhanced Machine Learning (REML), by synthesizing the literature in various domains in ML with consistent notations which is missing from the current literature. The goal of this work is to equip researchers across various disciplines with a comprehensive, formally structured framework of retrieval-enhanced models, thereby fostering interdisciplinary future research.
  arXiv  Detail & Related papers  (2024-07-17T20:01:21Z)
• Systematic Task Exploration with LLMs: A Study in Citation Text Generation [63.50597360948099]
  Large language models (LLMs) bring unprecedented flexibility in defining and executing complex, creative natural language generation (NLG) tasks. We propose a three-component research framework that consists of systematic input manipulation, reference data, and output measurement. We use this framework to explore citation text generation -- a popular scholarly NLP task that lacks consensus on the task definition and evaluation metric.
  arXiv  Detail & Related papers  (2024-07-04T16:41:08Z)
• MASSW: A New Dataset and Benchmark Tasks for AI-Assisted Scientific Workflows [58.56005277371235]
  We introduce MASSW, a comprehensive text dataset on Multi-Aspect Summarization of ScientificAspects. MASSW includes more than 152,000 peer-reviewed publications from 17 leading computer science conferences spanning the past 50 years. We demonstrate the utility of MASSW through multiple novel machine-learning tasks that can be benchmarked using this new dataset.
  arXiv  Detail & Related papers  (2024-06-10T15:19:09Z)
• Text Generation: A Systematic Literature Review of Tasks, Evaluation, and Challenges [7.140449861888235]
  This review categorizes works in text generation into five main tasks. For each task, we review their relevant characteristics, sub-tasks, and specific challenges. Our investigation shows nine prominent challenges common to all tasks and sub-tasks in recent text generation publications.
  arXiv  Detail & Related papers  (2024-05-24T14:38:11Z)
• LLMs with Industrial Lens: Deciphering the Challenges and Prospects -- A Survey [7.691230862439528]
  Large language models (LLMs) have become the secret ingredient driving numerous industrial applications.<n>This paper aims to unravel and evaluate the obstacles and opportunities inherent in leveraging LLMs within an industrial context.
  arXiv  Detail & Related papers  (2024-02-22T13:52:02Z)
• Identifying Concerns When Specifying Machine Learning-Enabled Systems: A Perspective-Based Approach [1.2184324428571227]
  PerSpecML is a perspective-based approach for specifying ML-enabled systems. It helps practitioners identify which attributes, including ML and non-ML components, are important to contribute to the overall system's quality.
  arXiv  Detail & Related papers  (2023-09-14T18:31:16Z)
• A Comprehensive Overview of Large Language Models [68.22178313875618]
  Large Language Models (LLMs) have recently demonstrated remarkable capabilities in natural language processing tasks. This article provides an overview of the existing literature on a broad range of LLM-related concepts.
  arXiv  Detail & Related papers  (2023-07-12T20:01:52Z)
• A Survey on Multimodal Large Language Models [71.63375558033364]
  Multimodal Large Language Model (MLLM) represented by GPT-4V has been a new rising research hotspot. This paper aims to trace and summarize the recent progress of MLLMs.
  arXiv  Detail & Related papers  (2023-06-23T15:21:52Z)
• Mapping the Challenges of HCI: An Application and Evaluation of ChatGPT and GPT-4 for Mining Insights at Scale [4.6542291555324296]
  We evaluate the combination of ChatGPT and GPT-4 for the real-world task of mining insights from a text corpus. We extract 4,392 research challenges in over 100 topics from the 2023CHI conference proceedings. We conclude that the combination of ChatGPT and GPT-4 makes an excellent cost-efficient means for analyzing a text corpus at scale.
  arXiv  Detail & Related papers  (2023-06-08T08:41:30Z)
• More Engineering, No Silos: Rethinking Processes and Interfaces in Collaboration between Interdisciplinary Teams for Machine Learning Projects [4.482886054198202]
  We identify key collaboration challenges that teams face when building and deploying machine learning systems into production. We report on common collaboration points in the development of production ML systems for requirements, data, and integration, as well as corresponding team patterns and challenges.
  arXiv  Detail & Related papers  (2021-10-19T20:03:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.