Quality and Security Signals in AI-Generated Python Refactoring Pull Requests

• URL: http://arxiv.org/abs/2605.21453v1
• Date: Wed, 20 May 2026 17:43:36 GMT
• Title: Quality and Security Signals in AI-Generated Python Refactoring Pull Requests
• Abstract summary: We conduct an empirical study of Python pull requests (PRs) from the AIDev dataset.<n>We analyze agentic PRs using PyQu, an ML-based quality assessment tool for Python.<n>Our results show that, on average, agentic commits improve a quality attribute in 22.5% of the studied changes.<n>At the same time, 24.17% of modified files introduce new Pylint issues predominantly convention level violations.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As AI agents increasingly contribute to code development and maintenance, there is still limited empirical evidence on the quality and risk characteristics of their changes in real-world projects, particularly for refactoring-oriented contributions. It remains unclear how agent-authored refactoring edits affect maintainability, code quality, and security once merged into GitHub repositories. To address this gap, we conduct an empirical study of Python refactoring pull requests (PRs) from the AIDev dataset. We analyze agentic refactoring PRs using PyQu, an ML-based quality assessment tool for Python, to quantify changes across five quality attributes, and we complement PyQu with domain-independent static analysis (Pylint and Bandit) to measure code quality and security issues before and after each change. Our results show that, on average, agentic commits improve a quality attribute in 22.5% of the studied changes, with usability improving most frequently (36.5%). At the same time, 24.17% of modified files introduce new Pylint issues predominantly convention level violations such as long lines-while 4.7% introduce new Bandit findings. From the observed diffs, we derive a taxonomy of 24 recurring change operations and map them to the lint and security findings they most commonly affect. Despite these mixed outcomes, developer acceptance is high: 73.5% of the analyzed PRs are merged, including cases that introduce new lint or security findings, often alongside the removal of existing issues. Overall, these findings highlight both the promise and current limitations of agentic refactoring, and motivate stronger tool-in-the-loop quality and security gating for AI-driven development workflows.

Related papers

• "Refactoring Runaway": Understanding and Mitigating Tangled Refactorings in Coding Agents for Issue Resolution [12.987766332400213]
  We analyze 3,691 valid patches generated by three agent frameworks with 12 LLMs.<n>We find that coding agents introduce tangleds less frequently than human developers.<n>We propose a refinement approach that assesses the necessity and safety of tangleds and selectively removes problematic operations.
  arXiv  Detail & Related papers  (2026-05-21T14:18:29Z)
• ClawArena: Benchmarking AI Agents in Evolving Information Environments [61.664633997138004]
  ClawArena is a benchmark for evaluating AI agents in evolving information environments.<n>Each scenario maintains a complete hidden ground truth while exposing the agent only to noisy, partial, and sometimes contradictory traces.<n> evaluation is organized around three coupled challenges: multi-source conflict reasoning, dynamic belief revision, and implicit personalization.
  arXiv  Detail & Related papers  (2026-04-05T17:55:23Z)
• Safer Builders, Risky Maintainers: A Comparative Study of Breaking Changes in Human vs Agentic PRs [0.991629944808926]
  We conduct a comparative analysis of 7,191 agent-generated PRs with 1402 human-authored PRs from Python repositories.<n>We show that AI agents introduce fewer breaking changes overall than humans (3.45% vs. 7.40%) in code generation tasks.<n>Agents exhibit substantially higher risk during maintenance tasks, with changes introducing breaking changes at rates of 6.72% and 9.35%, respectively.
  arXiv  Detail & Related papers  (2026-03-29T05:15:13Z)
• SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks [55.76734816061826]
  We introduce SlopCodeBench, a language-agnostic benchmark comprising 20 problems and 93 checkpoints.<n>We track two trajectory-level quality signals: verbosity, the fraction of redundant or duplicated code, and structural erosion.<n>No agent solves any problem end-to-end across 11 models; the highest checkpoint solve rate is 17.2%.
  arXiv  Detail & Related papers  (2026-03-25T19:26:44Z)
• Do AI Agents Really Improve Code Readability? [0.0]
  Poor readability extends development time, increases bug-inducing risks, and contributes to technical debt.<n>With the rapid advancement of Large Language Models, AI agent-based approaches have emerged as a promising paradigm for automated decomposing complex tasks.<n>This study investigates the impact of AI agent-based specifically improves code readability.
  arXiv  Detail & Related papers  (2026-03-14T03:01:10Z)
• Beyond Bug Fixes: An Empirical Investigation of Post-Merge Code Quality Issues in Agent-Generated Pull Requests [4.744786007044749]
  We analyze 1,210 merged agent-generated bug-fix PRs from Python repositories in the AIDev dataset.<n>Our results show that apparent differences in raw issue counts across agents largely disappear after normalizing by code churn.<n>Across all agents, code smells dominate, particularly at critical and major severities, while bugs are less frequent but often severe.
  arXiv  Detail & Related papers  (2026-01-27T22:55:05Z)
• Agentic Refactoring: An Empirical Study of AI Coding Agents [9.698067623031909]
  Agentic coding tools, such as OpenAI Codex, Claude Code, and Cursor, are transforming the software engineering landscape.<n>These AI-powered systems function as autonomous teammates capable of planning and executing complex development tasks.<n>There is a critical lack of empirical understanding regarding how agentic is utilized in practice, how it compares to human-driven, and what impact it has on code quality.
  arXiv  Detail & Related papers  (2025-11-06T21:24:38Z)
• From Code Changes to Quality Gains: An Empirical Study in Python ML Systems with PyQu [1.5259434915412646]
  We introduce PyQu, a novel tool to identify quality-enhancing commits with an average accuracy, precision, and recall of 0.84 and 0.85 of average F1 score.<n> 41% of the changes are newly discovered by our study and have not been identified by state-of-the-art Python changes detection tools.
  arXiv  Detail & Related papers  (2025-11-04T18:55:19Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• Refactoring $\neq$ Bug-Inducing: Improving Defect Prediction with Code Change Tactics Analysis [54.361900378970134]
  Just-in-time defect prediction (JIT-DP) aims to predict the likelihood of code changes resulting in software defects at an early stage.<n>Prior research has largely ignored code during both the evaluation and methodology phases, despite its prevalence.<n>We propose Code chAnge Tactics (CAT) analysis to categorize code and its propagation, which improves labeling accuracy in the JIT-Defects4J dataset by 13.7%.
  arXiv  Detail & Related papers  (2025-07-25T23:29:25Z)
• ParamMute: Suppressing Knowledge-Critical FFNs for Faithful Retrieval-Augmented Generation [91.20492150248106]
  We investigate the internal mechanisms behind unfaithful generation and identify a subset of mid-to-deep feed-forward networks (FFNs) that are disproportionately activated in such cases.<n>We propose Parametric Knowledge Muting through FFN Suppression (ParamMute), a framework that improves contextual faithfulness by suppressing the activation of unfaithfulness-associated FFNs.<n> Experimental results show that ParamMute significantly enhances faithfulness across both CoFaithfulQA and the established ConFiQA benchmark, achieving substantial reductions in reliance on parametric memory.
  arXiv  Detail & Related papers  (2025-02-21T15:50:41Z)
• Leveraging Large Language Models for Efficient Failure Analysis in Game Development [47.618236610219554]
  This paper proposes a new approach to automatically identify which change in the code caused a test to fail. The method leverages Large Language Models (LLMs) to associate error messages with the corresponding code changes causing the failure. Our approach reaches an accuracy of 71% in our newly created dataset, which comprises issues reported by developers at EA over a period of one year.
  arXiv  Detail & Related papers  (2024-06-11T09:21:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.