How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversations

• URL: http://arxiv.org/abs/2402.06013v1
• Date: Thu, 8 Feb 2024 19:24:01 GMT
• Title: How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversations
• Authors: Eman Abdullah AlOmar, Anushkrishna Venkatakrishnan, Mohamed Wiem Mkaouer, Christian D. Newman, Ali Ouni
• Abstract summary: Little is known about how developers articulate their needs when interacting with ChatGPT. Our approach relies on text mining-related conversations from 17,913 ChatGPT prompts and responses. We envision that our findings contribute to a broader understanding of the collaboration between developers and AI models, in the context of code, with implications for model improvement, tool development, and best practices in software engineering.
• Score: 14.736597134188065
• License: http://creativecommons.org/publicdomain/zero/1.0/
• Abstract: Large Language Models (LLMs), like ChatGPT, have gained widespread popularity and usage in various software engineering tasks, including refactoring, testing, code review, and program comprehension. Despite recent studies delving into refactoring documentation in commit messages, issues, and code review, little is known about how developers articulate their refactoring needs when interacting with ChatGPT. In this paper, our goal is to explore conversations between developers and ChatGPT related to refactoring to better understand how developers identify areas for improvement in code and how ChatGPT addresses developers' needs. Our approach relies on text mining refactoring-related conversations from 17,913 ChatGPT prompts and responses, and investigating developers' explicit refactoring intention. Our results reveal that (1) developer-ChatGPT conversations commonly involve generic and specific terms/phrases; (2) developers often make generic refactoring requests, while ChatGPT typically includes the refactoring intention; and (3) various learning settings when prompting ChatGPT in the context of refactoring. We envision that our findings contribute to a broader understanding of the collaboration between developers and AI models, in the context of code refactoring, with implications for model improvement, tool development, and best practices in software engineering.

Related papers

• Automated Refactoring of Non-Idiomatic Python Code: A Differentiated Replication with LLMs [54.309127753635366]
  We present the results of a replication study in which we investigate GPT-4 effectiveness in recommending and suggesting idiomatic actions. Our findings underscore the potential of LLMs to achieve tasks where, in the past, implementing recommenders based on complex code analyses was required.
  arXiv  Detail & Related papers  (2025-01-28T15:41:54Z)
• Why Do Developers Engage with ChatGPT in Issue-Tracker? Investigating Usage and Reliance on ChatGPT-Generated Code [4.605779671279481]
  We analyzed 1,152 Developer-ChatGPT conversations across 1,012 issues in GitHub. ChatGPT is primarily utilized for ideation, whereas its usage for validation is minimal. ChatGPT-generated code was used as-is to resolve only 5.83% of the issues.
  arXiv  Detail & Related papers  (2024-12-09T18:47:31Z)
• Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study [2.930521532345053]
  We explore alternatives to ChatGPT in code refinement tasks by including two open-source, smaller-scale large language models: CodeLlama and Llama 2. Our results show that, if properly tuned, the Llama models can achieve reasonable performance, often comparable to ChatGPT in automated code refinement. Our study highlights the potential of open-source models for code refinement, offering cost-effective, privacy-conscious solutions for real-world software development.
  arXiv  Detail & Related papers  (2024-12-03T19:39:31Z)
• You Augment Me: Exploring ChatGPT-based Data Augmentation for Semantic Code Search [47.54163552754051]
  Code search plays a crucial role in software development, enabling developers to retrieve and reuse code using natural language queries. Recently, large language models (LLMs) have made remarkable progress in both natural and programming language understanding and generation. We propose a novel approach ChatDANCE, which utilizes high-quality and diverse augmented data generated by a large language model.
  arXiv  Detail & Related papers  (2024-08-10T12:51:21Z)
• CodeRAG-Bench: Can Retrieval Augment Code Generation? [78.37076502395699]
  We conduct a systematic, large-scale analysis of code generation using retrieval-augmented generation. We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks. We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources.
  arXiv  Detail & Related papers  (2024-06-20T16:59:52Z)
• An Empirical Study on Developers Shared Conversations with ChatGPT in GitHub Pull Requests and Issues [20.121332699827633]
  ChatGPT has significantly impacted software development practices. Despite its widespread adoption, the impact of ChatGPT as an assistant in collaborative coding remains largely unexplored. We analyze a dataset of 210 and 370 developers shared conversations with ChatGPT in GitHub pull requests (PRs) and issues.
  arXiv  Detail & Related papers  (2024-03-15T16:58:37Z)
• ChatDev: Communicative Agents for Software Development [84.90400377131962]
  ChatDev is a chat-powered software development framework in which specialized agents are guided in what to communicate. These agents actively contribute to the design, coding, and testing phases through unified language-based communication.
  arXiv  Detail & Related papers  (2023-07-16T02:11:34Z)
• Improving ChatGPT Prompt for Code Generation [13.303599826870705]
  OpenAI's language model ChatGPT has emerged as a powerful tool for generating human-like responses to a wide range of textual inputs. We evaluate ChatGPT's capabilities for two code generation tasks, including text-to-code and code-to-code generation. Our results showed that by carefully designing prompts to guide ChatGPT, the generation performance can be improved substantially.
  arXiv  Detail & Related papers  (2023-05-15T05:37:33Z)
• Do code refactorings influence the merge effort? [80.1936417993664]
  Multiple contributors frequently change the source code in parallel to implement new features, fix bugs, existing code, and make other changes. These simultaneous changes need to be merged into the same version of the source code. Studies show that 10 to 20 percent of all merge attempts result in conflicts, which require the manual developer's intervention to complete the process.
  arXiv  Detail & Related papers  (2023-05-10T13:24:59Z)
• Reasoning in Dialog: Improving Response Generation by Context Reading Comprehension [49.92173751203827]
  In multi-turn dialog, utterances do not always take the full form of sentences. We propose to improve the response generation performance by examining the model's ability to answer a reading comprehension question.
  arXiv  Detail & Related papers  (2020-12-14T10:58:01Z)
• How We Refactor and How We Document it? On the Use of Supervised Machine Learning Algorithms to Classify Refactoring Documentation [25.626914797750487]
  Refactoring is the art of improving the design of a system without altering its external behavior. This study categorizes commits into 3 categories, namely, Internal QA, External QA, and Code Smell Resolution, along with the traditional BugFix and Functional categories. To better understand our classification results, we analyzed commit messages to extract patterns that developers regularly use to describe their smells.
  arXiv  Detail & Related papers  (2020-10-26T20:33:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.