Patterns of Developer Adoption of LLM-Generated Code Refactoring Suggestions

• URL: http://arxiv.org/abs/2605.04835v1
• Date: Wed, 06 May 2026 12:31:49 GMT
• Title: Patterns of Developer Adoption of LLM-Generated Code Refactoring Suggestions
• Abstract summary: Large language models (LLMs) have gained widespread popularity and have steadily improved over time.<n>LLMs suggest changes for the developer to apply to their code to improve quality attributes such as readability or maintainability.<n>We analyze 169 GitHub commits where developers their code based on a ChatGPT conversation linked in the commit message.<n>We found that developers mostly accept and use the suggestions without modifications.
• Score: 3.008881106008445
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) have gained widespread popularity and have steadily improved over time, enabling software developers to use them for various code-related tasks. One common task is code refactoring, where the LLM suggests changes for the developer to apply to their code to improve quality attributes such as readability or maintainability. While current research focuses on evaluating LLM-generated refactoring suggestions, there is a limited understanding of how developers apply these suggestions in practice. To explore this, we analyze 169 GitHub commits where developers refactor their code based on a ChatGPT conversation linked in the commit message. We found that developers mostly accept and use the suggestions without modifications. When changes are made, they are mostly major and fall into five different patterns that depend on the refactoring activity, the developer's prompt, and the validity of the response from ChatGPT.

Related papers

• RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices [54.956760584923295]
  Researchers have made substantial progress using Large Language Models (LLMs) for code generation.<n>However, developers typically write code based on structured designs or specifications rather than raw natural language descriptions.<n>This gap between existing benchmarks and real industry development practices means that current benchmark scores may not accurately reflect how much code generation can help automate development tasks.
  arXiv  Detail & Related papers  (2026-04-24T15:35:54Z)
• CodeTaste: Can LLMs Generate Human-Level Code Refactorings? [2.447746234944228]
  Large language model (LLM) coding agents can generate working code, but their solutions often accumulate complexity, duplication, and architectural debt.<n>Human developers address such issues through: behavior-preserving program that improve structure and maintainability.<n>We present CodeTaste, a benchmark of tasks mined from large-scale multi-file changes in open-source repositories.
  arXiv  Detail & Related papers  (2026-03-04T15:34:18Z)
• How do Agents Refactor: An Empirical Study [2.7711196026307476]
  We present the first analysis of agentic pull requests in Java.<n>We identify types and detect code smells before and after commits.<n>We find Cursor to be the only model to show a statistically significant increase in smells.
  arXiv  Detail & Related papers  (2026-01-28T01:34:15Z)
• ChatGPT for Code Refactoring: Analyzing Topics, Interaction, and Effective Prompts [10.31253409274086]
  Large Language Models (LLMs) have become widely used in various software engineering tasks such as testing, code review, and program comprehension.<n>Our goal is to explore interactions related to between developers and ChatGPT to better understand how developers identify areas for improvement in code.<n>Our approach involves text mining 715-related interactions from 29,778 ChatGPT prompts and responses, as well as the analysis of developers' explicit intentions.
  arXiv  Detail & Related papers  (2025-09-09T18:55:03Z)
• Turning the Tide: Repository-based Code Reflection [52.13709676656648]
  We introduce LiveRepoReflection, a benchmark for evaluating code understanding and generation in multi-file repository contexts.<n>1,888 rigorously filtered test cases across $6$ programming languages to ensure diversity, correctness, and high difficulty.<n>We also create RepoReflection-Instruct, a large-scale, quality-filtered instruction-tuning dataset derived from diverse sources.
  arXiv  Detail & Related papers  (2025-07-14T02:36:27Z)
• CodeRAG-Bench: Can Retrieval Augment Code Generation? [78.37076502395699]
  We conduct a systematic, large-scale analysis of code generation using retrieval-augmented generation.<n>We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks.<n>We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources.
  arXiv  Detail & Related papers  (2024-06-20T16:59:52Z)
• Comments as Natural Logic Pivots: Improve Code Generation via Comment Perspective [85.48043537327258]
  We propose MANGO (comMents As Natural loGic pivOts), including a comment contrastive training strategy and a corresponding logical comment decoding strategy. Results indicate that MANGO significantly improves the code pass rate based on the strong baselines. The robustness of the logical comment decoding strategy is notably higher than the Chain-of-thoughts prompting.
  arXiv  Detail & Related papers  (2024-04-11T08:30:46Z)
• CodeEditorBench: Evaluating Code Editing Capability of Large Language Models [49.387195629660994]
  Large Language Models (LLMs) for code are rapidly evolving, with code editing emerging as a critical capability.<n>We introduce CodeEditorBench, an evaluation framework designed to rigorously assess the performance of LLMs in code editing tasks.<n>We curate diverse coding challenges and scenarios from five sources, covering various programming languages, complexity levels, and editing tasks.
  arXiv  Detail & Related papers  (2024-04-04T15:49:49Z)
• How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversations [14.736597134188065]
  Little is known about how developers articulate their needs when interacting with ChatGPT. Our approach relies on text mining-related conversations from 17,913 ChatGPT prompts and responses. We envision that our findings contribute to a broader understanding of the collaboration between developers and AI models, in the context of code, with implications for model improvement, tool development, and best practices in software engineering.
  arXiv  Detail & Related papers  (2024-02-08T19:24:01Z)
• Do code refactorings influence the merge effort? [80.1936417993664]
  Multiple contributors frequently change the source code in parallel to implement new features, fix bugs, existing code, and make other changes. These simultaneous changes need to be merged into the same version of the source code. Studies show that 10 to 20 percent of all merge attempts result in conflicts, which require the manual developer's intervention to complete the process.
  arXiv  Detail & Related papers  (2023-05-10T13:24:59Z)
• How We Refactor and How We Document it? On the Use of Supervised Machine Learning Algorithms to Classify Refactoring Documentation [25.626914797750487]
  Refactoring is the art of improving the design of a system without altering its external behavior. This study categorizes commits into 3 categories, namely, Internal QA, External QA, and Code Smell Resolution, along with the traditional BugFix and Functional categories. To better understand our classification results, we analyzed commit messages to extract patterns that developers regularly use to describe their smells.
  arXiv  Detail & Related papers  (2020-10-26T20:33:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.