Operationalizing AI: Empirical Evidence on MLOps Practices, User Satisfaction, and Organizational Context

• URL: http://arxiv.org/abs/2510.09968v1
• Date: Sat, 11 Oct 2025 02:57:14 GMT
• Title: Operationalizing AI: Empirical Evidence on MLOps Practices, User Satisfaction, and Organizational Context
• Abstract summary: This study analyzes over 8,000 user reviews of AI development platforms from G2.com.<n>We measure review sentiment toward nine established Machine Learning Operations practices.<n>Seven of the nine practices show a significant positive relationship with user satisfaction.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Organizational efforts to utilize and operationalize artificial intelligence (AI) are often accompanied by substantial challenges, including scalability, maintenance, and coordination across teams. In response, the concept of Machine Learning Operations (MLOps) has emerged as a set of best practices that integrate software engineering principles with the unique demands of managing the ML lifecycle. Yet, empirical evidence on whether and how these practices support users in developing and operationalizing AI applications remains limited. To address this gap, this study analyzes over 8,000 user reviews of AI development platforms from G2.com. Using zero-shot classification, we measure review sentiment toward nine established MLOps practices, including continuous integration and delivery (CI/CD), workflow orchestration, reproducibility, versioning, collaboration, and monitoring. Seven of the nine practices show a significant positive relationship with user satisfaction, suggesting that effective MLOps implementation contributes tangible value to AI development. However, organizational context also matters: reviewers from small firms discuss certain MLOps practices less frequently, suggesting that organizational context influences the prevalence and salience of MLOps, though firm size does not moderate the MLOps-satisfaction link. This indicates that once applied, MLOps practices are perceived as universally beneficial across organizational settings.

Related papers

• Understanding How Enterprises Adopt the Model Context Protocol for LLM-Driven Software Engineering [4.345394092402742]
  Large Language Models (LLMs) are increasingly used in AI-based software engineering, but their limitations in complex task execution and multi-tool coordination have driven growing interest in the Model Context Protocol (MCP)<n>Existing research has mainly focused on MCP's technical design, with limited empirical evidence on how it is adopted and used in enterprise practice.<n>To address this gap, we conducted semi-structured interviews with 20 practitioners from eight companies in the Internet and financial sectors.
  arXiv  Detail & Related papers  (2026-06-08T08:20:50Z)
• Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind [8.740788873949471]
  Large Language Models (LLMs) have developed rapidly and are widely applied to both general-purpose and professional tasks.<n>They still struggle to comprehend and respond to the true user needs when intentions and instructions are imprecisely conveyed.
  arXiv  Detail & Related papers  (2026-02-14T16:01:59Z)
• Applying a Requirements-Focused Agile Management Approach for Machine Learning-Enabled Systems [1.3704574906282525]
  Machine Learning (ML)-enabled systems challenge traditional Requirements Engineering (RE) and agile management.<n>Existing RE and agile practices remain poorly integrated and insufficiently tailored to these characteristics.<n>This paper reports on the practical experience of applying RefineML, a requirements-focused approach for the continuous and agile refinement of ML-enabled systems.
  arXiv  Detail & Related papers  (2026-02-04T20:49:02Z)
• SelfAI: Building a Self-Training AI System with LLM Agents [79.10991818561907]
  SelfAI is a general multi-agent platform that combines a User Agent for translating high-level research objectives into standardized experimental configurations.<n>An Experiment Manager orchestrates parallel, fault-tolerant training across heterogeneous hardware while maintaining a structured knowledge base for continuous feedback.<n>Across regression, computer vision, scientific computing, medical imaging, and drug discovery benchmarks, SelfAI consistently achieves strong performance and reduces redundant trials.
  arXiv  Detail & Related papers  (2025-11-29T09:18:39Z)
• LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering [90.84806758077536]
  We introduce textbfLoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess large language models (LLMs) agents in realistic, long-context software engineering.<n>Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations.<n>Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens.
  arXiv  Detail & Related papers  (2025-11-17T23:57:24Z)
• EIFBENCH: Extremely Complex Instruction Following Benchmark for Large Language Models [64.70546873396624]
  We present the Extremely Complex Instruction Following Benchmark (EIFBENCH) for evaluating large language models (LLMs)<n>EIFBENCH includes multi-task scenarios that enable comprehensive assessment across diverse task types concurrently.<n>We also propose the Segment Policy Optimization (SegPO) algorithm to enhance the LLM's ability to accurately fulfill multi-task workflow.
  arXiv  Detail & Related papers  (2025-06-10T02:39:55Z)
• ORMind: A Cognitive-Inspired End-to-End Reasoning Framework for Operations Research [56.961539386979354]
  We introduce ORMind, a cognitive-inspired framework that enhances optimization through counterfactual reasoning.<n>Our approach emulates human cognition, implementing an end-to-end workflow that transforms requirements into mathematical models and executable code.<n>It is currently being tested internally in Lenovo's AI Assistant, with plans to enhance optimization capabilities for both business and consumer customers.
  arXiv  Detail & Related papers  (2025-06-02T05:11:21Z)
• Evaluations at Work: Measuring the Capabilities of GenAI in Use [28.124088786766965]
  Current AI benchmarks miss the messy, multi-turn nature of human-AI collaboration.<n>We present an evaluation framework that decomposes real-world tasks into interdependent subtasks.
  arXiv  Detail & Related papers  (2025-05-15T23:06:23Z)
• Model Utility Law: Evaluating LLMs beyond Performance through Mechanism Interpretable Metric [99.56567010306807]
  Large Language Models (LLMs) have become indispensable across academia, industry, and daily applications.<n>One core challenge of evaluation in the large language model (LLM) era is the generalization issue.<n>We propose Model Utilization Index (MUI), a mechanism interpretability enhanced metric that complements traditional performance scores.
  arXiv  Detail & Related papers  (2025-04-10T04:09:47Z)
• Fine-Grained Appropriate Reliance: Human-AI Collaboration with a Multi-Step Transparent Decision Workflow for Complex Task Decomposition [14.413413322901409]
  We propose to investigate the impact of a novel Multi-Step Transparent (MST) decision workflow on user reliance behaviors.<n>Our findings demonstrate that human-AI collaboration with an MST decision workflow can outperform one-step collaboration in specific contexts.<n>Our work highlights that there is no one-size-fits-all decision workflow that can help obtain optimal human-AI collaboration.
  arXiv  Detail & Related papers  (2025-01-19T01:03:09Z)
• Evaluating Cultural and Social Awareness of LLM Web Agents [113.49968423990616]
  We introduce CASA, a benchmark designed to assess large language models' sensitivity to cultural and social norms.<n>Our approach evaluates LLM agents' ability to detect and appropriately respond to norm-violating user queries and observations.<n>Experiments show that current LLMs perform significantly better in non-agent environments.
  arXiv  Detail & Related papers  (2024-10-30T17:35:44Z)
• Initial Insights on MLOps: Perception and Adoption by Practitioners [9.777475640906404]
  MLOps (Machine Learning and Operations) guidelines have emerged as the principal reference in this field. Despite the introduction of MLOps guidelines, there is still a degree of skepticism surrounding their implementation. This study aims to provide deeper insight into MLOps and its impact on the next phase of innovation in machine learning.
  arXiv  Detail & Related papers  (2024-08-01T11:08:29Z)
• A Multivocal Review of MLOps Practices, Challenges and Open Issues [9.227450931458907]
  MLOps has emerged as a key solution to address many socio-technical challenges of bringing ML models to production.<n>Despite the utility of MLOps, an integrated body of knowledge regarding MLOps remains elusive due to the diversity of ML productionalization challenges it addresses.<n>This paper presents a Multivocal Literature Review that systematically analyzes a corpus of 150 peer-reviewed and 48 grey literature to synthesize a unified conceptualization of MLOps.
  arXiv  Detail & Related papers  (2024-06-14T05:47:13Z)
• Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration [83.4031923134958]
  Corex is a suite of novel general-purpose strategies that transform Large Language Models into autonomous agents. Inspired by human behaviors, Corex is constituted by diverse collaboration paradigms including Debate, Review, and Retrieve modes. We demonstrate that orchestrating multiple LLMs to work in concert yields substantially better performance compared to existing methods.
  arXiv  Detail & Related papers  (2023-09-30T07:11:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.