Socio-Technical Anti-Patterns in Building ML-Enabled Software: Insights from Leaders on the Forefront

• URL: http://arxiv.org/abs/2607.03270v1
• Date: Fri, 03 Jul 2026 12:38:30 GMT
• Title: Socio-Technical Anti-Patterns in Building ML-Enabled Software: Insights from Leaders on the Forefront
• Abstract summary: We focus on socio-technical challenges of productionizing machine learning models centered around and within teams.<n>By analyzing talks from practitioners for practitioners of a community with over 11,000 members in their Slack workspace, we found 17 anti-patterns.<n>We list recommendations to overcome these problems, ranging from technical solutions over guidelines to organizational restructuring.
• Score: 4.692400531340393
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Although machine learning (ML)-enabled software systems seem to be a success story considering their rise in economic power, there are consistent reports from companies and practitioners struggling to bring ML models into production. Many papers have focused on specific, and purely technical aspects, such as testing and pipelines, but only few on socio-technical aspects. Driven by numerous anecdotes and reports from practitioners, our goal is to collect and analyze socio-technical challenges of productionizing ML models centered around and within teams. To this end, we conducted the largest qualitative empirical study in this area, involving the manual analysis of 66 hours of talks that have been recorded by the MLOps community. By analyzing talks from practitioners for practitioners of a community with over 11,000 members in their Slack workspace, we found 17 anti-patterns, often rooted in organizational or management problems. We further list recommendations to overcome these problems, ranging from technical solutions over guidelines to organizational restructuring. Finally, we contextu-alize our findings with previous research, confirming existing results, validating our own, and highlighting new insights.

Related papers

• Results-Actionability Gap: Understanding How Practitioners Evaluate LLM Products in the Wild [33.360335897400965]
  We identify ten evaluation practices spanning informal 'vibe checks' to organizational meta-work.<n>We introduce a novel fifth we call the results-actionability gap, in which practitioners gather evaluation data but cannot translate findings into concrete improvements.<n>Our analysis suggests these interpretive practices are necessary adaptations to LLM characteristics rather than methodological failures.
  arXiv  Detail & Related papers  (2026-01-25T10:36:59Z)
• Measuring what Matters: Construct Validity in Large Language Model Benchmarks [103.53142193393931]
  evaluating large language models (LLMs) is crucial for both assessing their capabilities and identifying safety or robustness issues prior to deployment.<n>We conduct a systematic review of 445 benchmarks from leading conferences in natural language processing and machine learning.<n>We find patterns related to the measured phenomena, tasks, and scoring metrics which undermine the validity of the resulting claims.
  arXiv  Detail & Related papers  (2025-11-03T17:39:40Z)
• Software Testing with Large Language Models: An Interview Study with Practitioners [2.198430261120653]
  The use of large language models in software testing is growing fast as they support numerous tasks.<n>However, their adoption often relies on informal experimentation rather than structured guidance.<n>This study investigates how software testing professionals use LLMs in practice to propose a preliminary, practitioner-informed guideline.
  arXiv  Detail & Related papers  (2025-10-20T05:06:56Z)
• A Survey of Vibe Coding with Large Language Models [93.88284590533242]
  "Vibe Coding" is a development methodology where developers validate AI-generated implementations through outcome observation.<n>Despite its transformative potential, the effectiveness of this emergent paradigm remains under-explored.<n>This survey provides the first comprehensive and systematic review of Vibe Coding with large language models.
  arXiv  Detail & Related papers  (2025-10-14T11:26:56Z)
• A Comprehensive Survey on Benchmarks and Solutions in Software Engineering of LLM-Empowered Agentic System [56.40989626804489]
  This survey provides the first holistic analysis of Large Language Models-powered software engineering.<n>We review over 150 recent papers and propose a taxonomy along two key dimensions: (1) Solutions, categorized into prompt-based, fine-tuning-based, and agent-based paradigms, and (2) Benchmarks, including tasks such as code generation, translation, and repair.
  arXiv  Detail & Related papers  (2025-10-10T06:56:50Z)
• How Do Communities of ML-Enabled Systems Smell? A Cross-Sectional Study on the Prevalence of Community Smells [13.840177755312665]
  We conducted an empirical study on 188 repositories from the NICHE dataset using the CADOCS tool to identify and analyze community smells.<n>We found that certain smells, such as Prima Donna Effects and Sharing Villainy, are more prevalent and fluctuate over time compared to others like Radio Silence or Organizational Skirmish.
  arXiv  Detail & Related papers  (2025-04-24T10:23:37Z)
• MLRC-Bench: Can Language Agents Solve Machine Learning Research Challenges? [66.87201770167012]
  MLRC-Bench is a benchmark designed to quantify how effectively language agents can tackle challenging Machine Learning (ML) Research Competitions.<n>Unlike prior work, MLRC-Bench measures the key steps of proposing and implementing novel research methods.<n>Even the best-performing tested agent closes only 9.3% of the gap between baseline and top human participant scores.
  arXiv  Detail & Related papers  (2025-04-13T19:35:43Z)
• Instruction Tuning for Large Language Models: A Survey [52.86322823501338]
  We make a systematic review of the literature, including the general methodology of supervised fine-tuning (SFT)<n>We also review the potential pitfalls of SFT along with criticism against it, along with efforts pointing out current deficiencies of existing strategies.
  arXiv  Detail & Related papers  (2023-08-21T15:35:16Z)
• Building Emotional Support Chatbots in the Era of LLMs [64.06811786616471]
  We introduce an innovative methodology that synthesizes human insights with the computational prowess of Large Language Models (LLMs) By utilizing the in-context learning potential of ChatGPT, we generate an ExTensible Emotional Support dialogue dataset, named ExTES. Following this, we deploy advanced tuning techniques on the LLaMA model, examining the impact of diverse training strategies, ultimately yielding an LLM meticulously optimized for emotional support interactions.
  arXiv  Detail & Related papers  (2023-08-17T10:49:18Z)
• Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies [104.32199881187607]
  Large language models (LLMs) have demonstrated remarkable performance across a wide array of NLP tasks. A promising approach to rectify these flaws is self-correction, where the LLM itself is prompted or guided to fix problems in its own output. This paper presents a comprehensive review of this emerging class of techniques.
  arXiv  Detail & Related papers  (2023-08-06T18:38:52Z)
• Towards machine learning guided by best practices [0.0]
  Machine learning (ML) is being used in software systems with multiple application fields, from medicine to software engineering (SE) This thesis aims to answer research questions that help to understand the practices used and discussed by practitioners and researchers in the SE community.
  arXiv  Detail & Related papers  (2023-04-29T10:58:37Z)
• A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners [13.565678193855609]
  Incorporating machine learning (ML) components into software products raises new software-engineering challenges. We collect 50 relevant papers that together interacted with over 4758 practitioners using guidelines for systematic literature reviews. We then collected, grouped, and organized the over 500 mentions of challenges within those papers.
  arXiv  Detail & Related papers  (2023-03-31T19:00:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.