From Code Changes to Quality Gains: An Empirical Study in Python ML Systems with PyQu

• URL: http://arxiv.org/abs/2511.02827v1
• Date: Tue, 04 Nov 2025 18:55:19 GMT
• Title: From Code Changes to Quality Gains: An Empirical Study in Python ML Systems with PyQu
• Abstract summary: We introduce PyQu, a novel tool to identify quality-enhancing commits with an average accuracy, precision, and recall of 0.84 and 0.85 of average F1 score.<n> 41% of the changes are newly discovered by our study and have not been identified by state-of-the-art Python changes detection tools.
• Score: 1.5259434915412646
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: In an era shaped by Generative Artificial Intelligence for code generation and the rising adoption of Python-based Machine Learning systems (MLS), software quality has emerged as a major concern. As these systems grow in complexity and importance, a key obstacle lies in understanding exactly how specific code changes affect overall quality-a shortfall aggravated by the lack of quality assessment tools and a clear mapping between ML systems code changes and their quality effects. Although prior work has explored code changes in MLS, it mostly stops at what the changes are, leaving a gap in our knowledge of the relationship between code changes and the MLS quality. To address this gap, we conducted a large-scale empirical study of 3,340 open-source Python ML projects, encompassing more than 3.7 million commits and 2.7 trillion lines of code. We introduce PyQu, a novel tool that leverages low level software metrics to identify quality-enhancing commits with an average accuracy, precision, and recall of 0.84 and 0.85 of average F1 score. Using PyQu and a thematic analysis, we identified 61 code changes, each demonstrating a direct impact on enhancing software quality, and we classified them into 13 categories based on contextual characteristics. 41% of the changes are newly discovered by our study and have not been identified by state-of-the-art Python changes detection tools. Our work offers a vital foundation for researchers, practitioners, educators, and tool developers, advancing the quest for automated quality assessment and best practices in Python-based ML software.

Related papers

• On the Relation between Code Quality and Machine Learning Performance: A Large-scale Empirical Study [0.13381749415517016]
  We investigated the relationship between code quality and machine learning (ML) performance in notebooks.<n>General Python code quality is decoupled from ML performance, showing negligible or non-significant correlations across all observations.<n>Code expertise provides no information on quality or performance, but competition expertise correlates with better performance.
  arXiv  Detail & Related papers  (2026-09-08T12:52:54Z)
• Quality and Security Signals in AI-Generated Python Refactoring Pull Requests [0.0]
  We conduct an empirical study of Python pull requests (PRs) from the AIDev dataset.<n>We analyze agentic PRs using PyQu, an ML-based quality assessment tool for Python.<n>Our results show that, on average, agentic commits improve a quality attribute in 22.5% of the studied changes.<n>At the same time, 24.17% of modified files introduce new Pylint issues predominantly convention level violations.
  arXiv  Detail & Related papers  (2026-05-20T17:43:36Z)
• FrontierSmith: Synthesizing Open-Ended Coding Problems at Scale [73.57353294235223]
  We introduce FrontierSmith, an automated system for evolving open-ended problems from existing closed-ended coding tasks.<n>On two open-ended coding benchmarks, training on our synthesized data yields substantial gains over the base models.<n>The synthesized problems also make agents take more turns and use more tokens, similar to human-curated ones.
  arXiv  Detail & Related papers  (2026-05-14T06:39:42Z)
• SimpleDevQA: Benchmarking Large Language Models on Development Knowledge QA [58.75982433502236]
  The Dev Knowledge QA task accounts for 39.6% of interactions.<n>Only 27.5% of real Dev Knowledge QA dialogues focus on code understanding.<n>Only 17.1% of real-world Dev Knowledge QA dialogues can be used for constructing a benchmark.
  arXiv  Detail & Related papers  (2025-12-09T17:58:36Z)
• From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
  Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
  arXiv  Detail & Related papers  (2025-11-23T17:09:34Z)
• Analysing Python Machine Learning Notebooks with Moose [0.12249546377051435]
  Machine Learning (ML) code, particularly within notebooks, often exhibits lower quality compared to traditional software.<n>Bad practices arise at three distinct levels: general Python coding conventions, the organizational structure of the notebook itself, and ML-specific aspects such as API usage.<n>This paper introduces Vespucci Linter, a static analysis tool with multi-level capabilities, built on Moose.
  arXiv  Detail & Related papers  (2025-09-15T09:59:49Z)
• PyExamine A Comprehensive, UnOpinionated Smell Detection Tool for Python [5.669063174637433]
  PyExamine is an approach to code smell detection that operates across multiple levels of analysis.<n>The metrics span across all levels of code organization, from high-level architectural concerns to granular implementation details.<n>PyExamine achieved detection accuracy rates: 91.4% for code-level smells, 89.3% for structural smells, and 80.6% for architectural smells.
  arXiv  Detail & Related papers  (2025-01-30T13:14:14Z)
• Understanding Code Understandability Improvements in Code Reviews [79.16476505761582]
  We analyzed 2,401 code review comments from Java open-source projects on GitHub. 83.9% of suggestions for improvement were accepted and integrated, with fewer than 1% later reverted.
  arXiv  Detail & Related papers  (2024-10-29T12:21:23Z)
• Leveraging Large Language Models for Efficient Failure Analysis in Game Development [47.618236610219554]
  This paper proposes a new approach to automatically identify which change in the code caused a test to fail. The method leverages Large Language Models (LLMs) to associate error messages with the corresponding code changes causing the failure. Our approach reaches an accuracy of 71% in our newly created dataset, which comprises issues reported by developers at EA over a period of one year.
  arXiv  Detail & Related papers  (2024-06-11T09:21:50Z)
• Towards Understanding the Impact of Code Modifications on Software Quality Metrics [1.2277343096128712]
  This study aims to assess and interpret the impact of code modifications on software quality metrics. The underlying hypothesis posits that code modifications inducing similar changes in software quality metrics can be grouped into distinct clusters. The results reveal distinct clusters of code modifications, each accompanied by a concise description, revealing their collective impact on software quality metrics.
  arXiv  Detail & Related papers  (2024-04-05T08:41:18Z)
• Bugs in Machine Learning-based Systems: A Faultload Benchmark [16.956588187947993]
  There is no standard benchmark of bugs to assess their performance, compare them and discuss their advantages and weaknesses. In this study, we firstly investigate the verifiability of the bugs in ML-based systems and show the most important factors in each one. We provide a benchmark namely defect4ML that satisfies all criteria of standard benchmark, i.e. relevance, fairness, verifiability, and usability.
  arXiv  Detail & Related papers  (2022-06-24T14:20:34Z)
• Uncertainty Quantification 360: A Holistic Toolkit for Quantifying and Communicating the Uncertainty of AI [49.64037266892634]
  We describe an open source Python toolkit named Uncertainty Quantification 360 (UQ360) for the uncertainty quantification of AI models. The goal of this toolkit is twofold: first, to provide a broad range of capabilities to streamline as well as foster the common practices of quantifying, evaluating, improving, and communicating uncertainty in the AI application development lifecycle; second, to encourage further exploration of UQ's connections to other pillars of trustworthy AI.
  arXiv  Detail & Related papers  (2021-06-02T18:29:04Z)
• Measuring Coding Challenge Competence With APPS [54.22600767666257]
  We introduce APPS, a benchmark for code generation. Our benchmark includes 10,000 problems, which range from having simple one-line solutions to being substantial algorithmic challenges. Recent models such as GPT-Neo can pass approximately 15% of the test cases of introductory problems.
  arXiv  Detail & Related papers  (2021-05-20T17:58:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.