How are MLOps Frameworks Used in Open Source Projects? An Empirical Characterization

• URL: http://arxiv.org/abs/2601.18591v1
• Date: Mon, 26 Jan 2026 15:35:42 GMT
• Title: How are MLOps Frameworks Used in Open Source Projects? An Empirical Characterization
• Authors: Fiorella Zampetti, Federico Stocchetti, Federica Razzano, Damian Andrew Tamburri, Massimiliano Di Penta,
• Abstract summary: Machine Learning (ML) Operations (MLOps) frameworks have been conceived to support developers and AI engineers in managing the lifecycle of their ML models.<n>This paper investigates the practical use and desired feature enhancements of eight popular open-source MLOps frameworks.
• Score: 7.351668608270923
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine Learning (ML) Operations (MLOps) frameworks have been conceived to support developers and AI engineers in managing the lifecycle of their ML models. While such frameworks provide a wide range of features, developers may leverage only a subset of them, while missing some highly desired features. This paper investigates the practical use and desired feature enhancements of eight popular open-source MLOps frameworks. Specifically, we analyze their usage by dependent projects on GitHub, examining how they invoke the frameworks' APIs and commands. Then, we qualitatively analyze feature requests and enhancements mined from the frameworks' issue trackers, relating these desired improvements to the previously identified usage features. Results indicate that MLOps frameworks are rarely used out-of-the-box and are infrequently integrated into GitHub Workflows, but rather, developers use their APIs to implement custom functionality in their projects. Used features concern core ML phases and whole infrastructure governance, sometimes leveraging multiple frameworks with complementary features. The mapping with feature requests highlights that users mainly ask for enhancements to core features of the frameworks, but also better API exposure and CI/CD integration.

Related papers

• ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development [72.4729759618632]
  We introduce ABC-Bench, a benchmark to evaluate agentic backend coding within a realistic, executable workflow.<n>We curated 224 practical tasks spanning 8 languages and 19 frameworks from open-source repositories.<n>Our evaluation reveals that even state-of-the-art models struggle to deliver reliable performance on these holistic tasks.
  arXiv  Detail & Related papers  (2026-01-16T08:23:52Z)
• Framework-Aware Code Generation with API Knowledge Graph-Constructed Data: A Study on HarmonyOS [52.483888557864326]
  APIKG4SYN is a framework designed to exploit API knowledge graphs for the construction of API-oriented question-code pairs.<n>We build the first benchmark for HarmonyOS code generation using APIKG4SYN.
  arXiv  Detail & Related papers  (2025-11-29T08:13:54Z)
• OpenLambdaVerse: A Dataset and Analysis of Open-Source Serverless Applications [0.6215404942415159]
  OpenLambdaVerse is a dataset of GitHub repositories that use the Serverless Framework in applications that contain one or more Lambda functions.<n>We gain important insights on the size and complexity of current applications, which languages and languages they employ, how are the functions triggered, the maturity of the projects, and their security practices.
  arXiv  Detail & Related papers  (2025-08-02T21:30:01Z)
• DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generation [31.237236649603123]
  Multimodal Large Language Models (MLLMs) have demonstrated remarkable capabilities in automated front-end engineering.<n>DesignBench is a benchmark for assessing MLLMs' capabilities in automated front-end engineering.
  arXiv  Detail & Related papers  (2025-06-06T17:21:21Z)
• LLM-Generated Microservice Implementations from RESTful API Definitions [3.740584607001637]
  This paper presents a system that uses Large Language Models (LLMs) to automate the API-first development of software.<n>The system generates OpenAPI specification, generating server code from it, and refining the code through a feedback loop that analyzes execution logs and error messages.<n>The system has the potential to benefit software developers, architects, and organizations to speed up software development cycles.
  arXiv  Detail & Related papers  (2025-02-13T20:50:33Z)
• Lightweight Syntactic API Usage Analysis with UCov [0.0]
  We present a novel conceptual framework designed to assist library maintainers in understanding the interactions allowed by their APIs. These customizable models enable library maintainers to improve their design ahead of release, reducing friction during evolution. We implement these models for Java libraries in a new tool UCov and demonstrate its capabilities on three libraries exhibiting diverse styles of interaction.
  arXiv  Detail & Related papers  (2024-02-19T10:33:41Z)
• Enhancing Open-Domain Task-Solving Capability of LLMs via Autonomous Tool Integration from GitHub [79.31134731122462]
  We introduce OpenAct benchmark to evaluate the open-domain task-solving capability, built on human expert consultation and repositories in GitHub.<n>We present OpenAgent, a novel LLM-based agent system that can tackle evolving queries in open domains through autonomously integrating specialized tools from GitHub.
  arXiv  Detail & Related papers  (2023-12-28T15:47:30Z)
• TaskWeaver: A Code-First Agent Framework [50.99683051759488]
  TaskWeaver is a code-first framework for building LLM-powered autonomous agents. It converts user requests into executable code and treats user-defined plugins as callable functions. It provides support for rich data structures, flexible plugin usage, and dynamic plugin selection.
  arXiv  Detail & Related papers  (2023-11-29T11:23:42Z)
• CRAFT: Customizing LLMs by Creating and Retrieving from Specialized Toolsets [75.64181719386497]
  We present CRAFT, a tool creation and retrieval framework for large language models (LLMs) It creates toolsets specifically curated for the tasks and equips LLMs with a component that retrieves tools from these sets to enhance their capability to solve complex tasks. Our method is designed to be flexible and offers a plug-and-play approach to adapt off-the-shelf LLMs to unseen domains and modalities, without any finetuning.
  arXiv  Detail & Related papers  (2023-09-29T17:40:26Z)
• Enhancing Architecture Frameworks by Including Modern Stakeholders and their Views/Viewpoints [48.87872564630711]
  The stakeholders with data science and Machine Learning related concerns, such as data scientists and data engineers, are yet to be included in existing architecture frameworks.<n>We surveyed 61 subject matter experts from over 25 organizations in 10 countries.
  arXiv  Detail & Related papers  (2023-08-09T21:54:34Z)
• Exploring the potential of flow-based programming for machine learning deployment in comparison with service-oriented architectures [8.677012233188968]
  We argue that part of the reason is infrastructure that was not designed for activities around data collection and analysis. We propose to consider flow-based programming with data streams as an alternative to commonly used service-oriented architectures for building software applications.
  arXiv  Detail & Related papers  (2021-08-09T15:06:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.