Detecting Behavioral Changes in Python Refactoring Implementations with Foundation Models

• URL: http://arxiv.org/abs/2608.09919v1
• Date: Mon, 10 Aug 2026 17:57:37 GMT
• Title: Detecting Behavioral Changes in Python Refactoring Implementations with Foundation Models
• Abstract summary: We propose an approach based on a foundation model that analyzes git-style diffs to identify behavioral changes introduced by Pythons.<n>Our model-based analysis uncovered distinct bugs among the seven types studied.<n>These results highlight the need to improve the robustness of current Python tools to ensure the correctness of automated code transformations.
• Score: 40.3747997962952
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Python is a widely adopted programming language, valued for its simplicity and flexibility. However, automated refactoring for Python remains challenging, even though refactoring is an essential practice in software evolution aimed at improving internal code structure without changing external behavior. Understanding how behavioral changes are introduced during refactoring is crucial, as such issues can compromise software reliability and reduce developer productivity. We propose an approach based on a foundation model oracle that analyzes git-style diffs to identify behavioral changes introduced by Python refactorings. We evaluated our technique on Rope refactoring implementations, reusing 1,152 refactoring attempts from a prior study and analyzing 217 resulting transformation pairs with the oracle. Our model-based analysis uncovered 13 distinct bugs among the seven refactoring types studied. All reported bugs were submitted to the respective developers, and 12 of the 13 resulting issue reports were accepted according to issue-tracker evidence. These results highlight the need to improve the robustness of current Python refactoring tools to ensure the correctness of automated code transformations and support reliable software maintenance.

Related papers

• From Human to Machine Refactoring: Assessing GPT-4's Impact on Python Class Quality and Readability [46.83143241367452]
  Refactoring aims to improve code quality without altering program behavior.<n>Recent advances in Large Language Models (LLMs) have introduced new opportunities for automated code preservation.<n>We present an empirical study on LLM-driven classes using GPT-4o, applied to 100 Python classes from the ClassEval benchmark.<n>Our findings show that GPT-4o generally produces behavior-preservings that reduce code smells and improve quality metrics, albeit at the cost of decreased readability.
  arXiv  Detail & Related papers  (2026-01-19T15:22:37Z)
• Code Refactoring with LLM: A Comprehensive Evaluation With Few-Shot Settings [0.0]
  This study aims to develop a framework capable of performing accurate and efficient code across languages (C, C++, C#, Python, Java)<n>Java achieves the highest overall correctness up to 99.99% the 10-shot setting, records the highest average compilability of 94.78% compared to the original source code.
  arXiv  Detail & Related papers  (2025-11-26T14:47:52Z)
• Agentic Refactoring: An Empirical Study of AI Coding Agents [9.698067623031909]
  Agentic coding tools, such as OpenAI Codex, Claude Code, and Cursor, are transforming the software engineering landscape.<n>These AI-powered systems function as autonomous teammates capable of planning and executing complex development tasks.<n>There is a critical lack of empirical understanding regarding how agentic is utilized in practice, how it compares to human-driven, and what impact it has on code quality.
  arXiv  Detail & Related papers  (2025-11-06T21:24:38Z)
• Refactoring $\neq$ Bug-Inducing: Improving Defect Prediction with Code Change Tactics Analysis [54.361900378970134]
  Just-in-time defect prediction (JIT-DP) aims to predict the likelihood of code changes resulting in software defects at an early stage.<n>Prior research has largely ignored code during both the evaluation and methodology phases, despite its prevalence.<n>We propose Code chAnge Tactics (CAT) analysis to categorize code and its propagation, which improves labeling accuracy in the JIT-Defects4J dataset by 13.7%.
  arXiv  Detail & Related papers  (2025-07-25T23:29:25Z)
• RefModel: Detecting Refactorings using Foundation Models [2.2670483018110366]
  We investigate the viability of using foundation models for detection, implemented in a tool named RefModel.<n>We evaluate Phi4-14B, and Claude 3.5 Sonnet on a dataset of 858 single-operation transformations applied to artificially generated Java programs.<n>In real-world settings, Claude 3.5 Sonnet and Gemini 2.5 Pro jointly identified 97% of all transformations, surpassing the best-performing static-analysis-based tools.
  arXiv  Detail & Related papers  (2025-07-15T14:20:56Z)
• Bugs in the Shadows: Static Detection of Faulty Python Refactorings [44.115219601924856]
  Python's dynamic type system poses significant challenges for automated code transformations.<n>Our analysis uncovered 29 bugs across four types from a total of 1,152 attempts.<n>These results highlight the need to improve the robustness of current Python tools to ensure the correctness of automated code transformations.
  arXiv  Detail & Related papers  (2025-07-01T18:03:56Z)
• ActRef: Enhancing the Understanding of Python Code Refactoring with Action-Based Analysis [10.724563250102696]
  This study presents an action-based Refactoring Analysis Framework named ActRef.<n>ActRef mining multiple types (e.g., move, rename, extract, and inline operations) based on diff actions.<n>By focusing on the code change actions, ActRef provides a Python-adaptive solution to detect intricate patterns.
  arXiv  Detail & Related papers  (2025-05-10T07:48:50Z)
• Automated Refactoring of Non-Idiomatic Python Code: A Differentiated Replication with LLMs [54.309127753635366]
  We present the results of a replication study in which we investigate GPT-4 effectiveness in recommending and suggesting idiomatic actions.<n>Our findings underscore the potential of LLMs to achieve tasks where, in the past, implementing recommenders based on complex code analyses was required.
  arXiv  Detail & Related papers  (2025-01-28T15:41:54Z)
• Do code refactorings influence the merge effort? [80.1936417993664]
  Multiple contributors frequently change the source code in parallel to implement new features, fix bugs, existing code, and make other changes. These simultaneous changes need to be merged into the same version of the source code. Studies show that 10 to 20 percent of all merge attempts result in conflicts, which require the manual developer's intervention to complete the process.
  arXiv  Detail & Related papers  (2023-05-10T13:24:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.