The role of library versions in Developer-ChatGPT conversations

• URL: http://arxiv.org/abs/2401.16340v1
• Date: Mon, 29 Jan 2024 17:46:18 GMT
• Title: The role of library versions in Developer-ChatGPT conversations
• Authors: Rachna Raj and Diego Elias Costa
• Abstract summary: We analyze DevGPT, a dataset of more than 4,000 Developer-ChatGPT interactions. We quantify how often library version constraints are mentioned in code-related conversations.
• Score: 1.6252311034292488
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The latest breakthroughs in large language models (LLM) have empowered software development tools, such as ChatGPT, to aid developers in complex tasks. Developers use ChatGPT to write code, review code changes, and even debug their programs. In these interactions, ChatGPT often recommends code snippets that depend on external libraries. However, code from libraries changes over time, invalidating a once-correct code snippet and making it difficult to reuse recommended code. In this study, we analyze DevGPT, a dataset of more than 4,000 Developer-ChatGPT interactions, to understand the role of library versions in code-related conversations. We quantify how often library version constraints are mentioned in code-related conversations and when ChatGPT recommends the installation of specific libraries. Our findings show that, albeit to constantly recommend and analyze code with external dependencies, library version constraints only appear in 9% of the conversations. In the majority of conversations, the version constraints are prompted by users (as opposed to being specified by ChatGPT) as a method for receiving better quality responses. Moreover, we study how library version constraints are used in the conversation through qualitative methods, identifying several potential problems that warrant further research.

Related papers

• Exploring the Potential of Llama Models in Automated Code Refinement: A Replication Study [2.930521532345053]
  We explore alternatives to ChatGPT in code refinement tasks by including two open-source, smaller-scale large language models: CodeLlama and Llama 2. Our results show that, if properly tuned, the Llama models can achieve reasonable performance, often comparable to ChatGPT in automated code refinement. Our study highlights the potential of open-source models for code refinement, offering cost-effective, privacy-conscious solutions for real-world software development.
  arXiv  Detail & Related papers  (2024-12-03T19:39:31Z)
• Commit0: Library Generation from Scratch [77.38414688148006]
  Commit0 is a benchmark that challenges AI agents to write libraries from scratch. Agents are provided with a specification document outlining the library's API as well as a suite of interactive unit tests. Commit0 also offers an interactive environment where models receive static analysis and execution feedback on the code they generate.
  arXiv  Detail & Related papers  (2024-12-02T18:11:30Z)
• Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendations [3.1911318265930944]
  ChatGPT uses third-party libraries nearly 10% more often than human developers. 14.2% of the recommended libraries had restrictive copyleft licenses, which were not explicitly communicated by ChatGPT. We recommend that developers implement rigorous dependency management practices and double-check library licenses before integrating LLM-generated code into their projects.
  arXiv  Detail & Related papers  (2024-08-09T15:36:59Z)
• CodeRAG-Bench: Can Retrieval Augment Code Generation? [78.37076502395699]
  We conduct a systematic, large-scale analysis of code generation using retrieval-augmented generation. We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks. We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources.
  arXiv  Detail & Related papers  (2024-06-20T16:59:52Z)
• Long Code Arena: a Set of Benchmarks for Long-Context Code Models [75.70507534322336]
  Long Code Arena is a suite of six benchmarks for code processing tasks that require project-wide context. These tasks cover different aspects of code processing: library-based code generation, CI builds repair, project-level code completion, commit message generation, bug localization, and module summarization. For each task, we provide a manually verified dataset for testing, an evaluation suite, and open-source baseline solutions.
  arXiv  Detail & Related papers  (2024-06-17T14:58:29Z)
• VersiCode: Towards Version-controllable Code Generation [58.82709231906735]
  Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development. We propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM) We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge.
  arXiv  Detail & Related papers  (2024-06-11T16:15:06Z)
• How to Refactor this Code? An Exploratory Study on Developer-ChatGPT Refactoring Conversations [14.736597134188065]
  Little is known about how developers articulate their needs when interacting with ChatGPT. Our approach relies on text mining-related conversations from 17,913 ChatGPT prompts and responses. We envision that our findings contribute to a broader understanding of the collaboration between developers and AI models, in the context of code, with implications for model improvement, tool development, and best practices in software engineering.
  arXiv  Detail & Related papers  (2024-02-08T19:24:01Z)
• Investigating the Utility of ChatGPT in the Issue Tracking System: An Exploratory Study [5.176434782905268]
  This study examines the interaction between ChatGPT and developers to analyze their prevalent activities and provide a resolution. Our investigation reveals that developers mainly use ChatGPT for brainstorming solutions but often opt to write their code instead of using ChatGPT-generated code.
  arXiv  Detail & Related papers  (2024-02-06T06:03:05Z)
• InterCode: Standardizing and Benchmarking Interactive Coding with Execution Feedback [50.725076393314964]
  We introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning environment. Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution. We demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies.
  arXiv  Detail & Related papers  (2023-06-26T17:59:50Z)
• Code Librarian: A Software Package Recommendation System [65.05559087332347]
  We present a recommendation engine called Librarian for open source libraries. A candidate library package is recommended for a given context if: 1) it has been frequently used with the imported libraries in the program; 2) it has similar functionality to the imported libraries in the program; 3) it has similar functionality to the developer's implementation, and 4) it can be used efficiently in the context of the provided code.
  arXiv  Detail & Related papers  (2022-10-11T12:30:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.