The role of library versions in Developer-ChatGPT conversations
- URL: http://arxiv.org/abs/2401.16340v1
- Date: Mon, 29 Jan 2024 17:46:18 GMT
- Title: The role of library versions in Developer-ChatGPT conversations
- Authors: Rachna Raj and Diego Elias Costa
- Abstract summary: We analyze DevGPT, a dataset of more than 4,000 Developer-ChatGPT interactions.
We quantify how often library version constraints are mentioned in code-related conversations.
- Score: 1.6252311034292488
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The latest breakthroughs in large language models (LLM) have empowered
software development tools, such as ChatGPT, to aid developers in complex
tasks. Developers use ChatGPT to write code, review code changes, and even
debug their programs. In these interactions, ChatGPT often recommends code
snippets that depend on external libraries. However, code from libraries
changes over time, invalidating a once-correct code snippet and making it
difficult to reuse recommended code.
In this study, we analyze DevGPT, a dataset of more than 4,000
Developer-ChatGPT interactions, to understand the role of library versions in
code-related conversations. We quantify how often library version constraints
are mentioned in code-related conversations and when ChatGPT recommends the
installation of specific libraries. Our findings show that, albeit to
constantly recommend and analyze code with external dependencies, library
version constraints only appear in 9% of the conversations. In the majority of
conversations, the version constraints are prompted by users (as opposed to
being specified by ChatGPT) as a method for receiving better quality responses.
Moreover, we study how library version constraints are used in the conversation
through qualitative methods, identifying several potential problems that
warrant further research.
Related papers
- Can OpenSource beat ChatGPT? -- A Comparative Study of Large Language Models for Text-to-Code Generation [0.24578723416255752]
We evaluate five different large language models (LLMs) concerning their capabilities for text-to-code generation.
ChatGPT can handle these typical programming challenges by far the most effectively, surpassing even code-specialized models like Code Llama.
arXiv Detail & Related papers (2024-09-06T10:03:49Z) - Is ChatGPT a Good Software Librarian? An Exploratory Study on the Use of ChatGPT for Software Library Recommendations [3.1911318265930944]
ChatGPT uses third-party libraries nearly 10% more often than human developers.
14.2% of the recommended libraries had restrictive copyleft licenses, which were not explicitly communicated by ChatGPT.
We recommend that developers implement rigorous dependency management practices and double-check library licenses before integrating LLM-generated code into their projects.
arXiv Detail & Related papers (2024-08-09T15:36:59Z) - CodeRAG-Bench: Can Retrieval Augment Code Generation? [78.37076502395699]
We conduct a systematic, large-scale analysis of code generation using retrieval-augmented generation.
We first curate a comprehensive evaluation benchmark, CodeRAG-Bench, encompassing three categories of code generation tasks.
We examine top-performing models on CodeRAG-Bench by providing contexts retrieved from one or multiple sources.
arXiv Detail & Related papers (2024-06-20T16:59:52Z) - Long Code Arena: a Set of Benchmarks for Long-Context Code Models [75.70507534322336]
Long Code Arena is a suite of six benchmarks for code processing tasks that require project-wide context.
These tasks cover different aspects of code processing: library-based code generation, CI builds repair, project-level code completion, commit message generation, bug localization, and module summarization.
For each task, we provide a manually verified dataset for testing, an evaluation suite, and open-source baseline solutions.
arXiv Detail & Related papers (2024-06-17T14:58:29Z) - VersiCode: Towards Version-controllable Code Generation [58.82709231906735]
Large Language Models (LLMs) have made tremendous strides in code generation, but existing research fails to account for the dynamic nature of software development.
We propose two novel tasks aimed at bridging this gap: version-specific code completion (VSCC) and version-aware code migration (VACM)
We conduct an extensive evaluation on VersiCode, which reveals that version-controllable code generation is indeed a significant challenge.
arXiv Detail & Related papers (2024-06-11T16:15:06Z) - How to Refactor this Code? An Exploratory Study on Developer-ChatGPT
Refactoring Conversations [14.736597134188065]
Little is known about how developers articulate their needs when interacting with ChatGPT.
Our approach relies on text mining-related conversations from 17,913 ChatGPT prompts and responses.
We envision that our findings contribute to a broader understanding of the collaboration between developers and AI models, in the context of code, with implications for model improvement, tool development, and best practices in software engineering.
arXiv Detail & Related papers (2024-02-08T19:24:01Z) - Investigating the Utility of ChatGPT in the Issue Tracking System: An
Exploratory Study [5.176434782905268]
This study examines the interaction between ChatGPT and developers to analyze their prevalent activities and provide a resolution.
Our investigation reveals that developers mainly use ChatGPT for brainstorming solutions but often opt to write their code instead of using ChatGPT-generated code.
arXiv Detail & Related papers (2024-02-06T06:03:05Z) - LILO: Learning Interpretable Libraries by Compressing and Documenting Code [71.55208585024198]
We introduce LILO, a neurosymbolic framework that iteratively synthesizes, compresses, and documents code.
LILO combines LLM-guided program synthesis with recent algorithmic advances in automated from Stitch.
We find that AutoDoc boosts performance by helping LILO's synthesizer to interpret and deploy learned abstractions.
arXiv Detail & Related papers (2023-10-30T17:55:02Z) - InterCode: Standardizing and Benchmarking Interactive Coding with
Execution Feedback [50.725076393314964]
We introduce InterCode, a lightweight, flexible, and easy-to-use framework of interactive coding as a standard reinforcement learning environment.
Our framework is language and platform agnostic, uses self-contained Docker environments to provide safe and reproducible execution.
We demonstrate InterCode's viability as a testbed by evaluating multiple state-of-the-art LLMs configured with different prompting strategies.
arXiv Detail & Related papers (2023-06-26T17:59:50Z) - Code Librarian: A Software Package Recommendation System [65.05559087332347]
We present a recommendation engine called Librarian for open source libraries.
A candidate library package is recommended for a given context if: 1) it has been frequently used with the imported libraries in the program; 2) it has similar functionality to the imported libraries in the program; 3) it has similar functionality to the developer's implementation, and 4) it can be used efficiently in the context of the provided code.
arXiv Detail & Related papers (2022-10-11T12:30:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.