Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward
- URL: http://arxiv.org/abs/2406.07765v1
- Date: Tue, 11 Jun 2024 23:10:43 GMT
- Title: Using AI-Based Coding Assistants in Practice: State of Affairs, Perceptions, and Ways Forward
- Authors: Agnia Sergeyuk, Yaroslav Golubev, Timofey Bryksin, Iftekhar Ahmed,
- Abstract summary: We carried out a large-scale survey aimed at how AI assistants are used.
We collected opinions of 481 programmers on five broad activities.
Our results show that usage of AI assistants varies depending on activity and stage.
- Score: 9.177785129949
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The last several years saw the emergence of AI assistants for code -- multi-purpose AI-based helpers in software engineering. Their quick development makes it necessary to better understand how specifically developers are using them, why they are not using them in certain parts of their development workflow, and what needs to be improved. In this work, we carried out a large-scale survey aimed at how AI assistants are used, focusing on specific software development activities and stages. We collected opinions of 481 programmers on five broad activities: (a) implementing new features, (b) writing tests, (c) bug triaging, (d) refactoring, and (e) writing natural-language artifacts, as well as their individual stages. Our results show that usage of AI assistants varies depending on activity and stage. For instance, developers find writing tests and natural-language artifacts to be the least enjoyable activities and want to delegate them the most, currently using AI assistants to generate tests and test data, as well as generating comments and docstrings most of all. This can be a good focus for features aimed to help developers right now. As for why developers do not use assistants, in addition to general things like trust and company policies, there are fixable issues that can serve as a guide for further research, e.g., the lack of project-size context, and lack of awareness about assistants. We believe that our comprehensive and specific results are especially needed now to steer active research toward where users actually need AI assistants.
Related papers
- The Ethics of Advanced AI Assistants [53.89899371095332]
This paper focuses on the opportunities and the ethical and societal risks posed by advanced AI assistants.
We define advanced AI assistants as artificial agents with natural language interfaces, whose function is to plan and execute sequences of actions on behalf of a user.
We consider the deployment of advanced assistants at a societal scale, focusing on cooperation, equity and access, misinformation, economic impact, the environment and how best to evaluate advanced AI assistants.
arXiv Detail & Related papers (2024-04-24T23:18:46Z) - Generating Java Methods: An Empirical Assessment of Four AI-Based Code
Assistants [5.32539007352208]
We assess the effectiveness of four popular AI-based code assistants, namely GitHub Copilot, Tabnine, ChatGPT, and Google Bard.
Results show that Copilot is often more accurate than other techniques, yet none of the assistants is completely subsumed by the rest of the approaches.
arXiv Detail & Related papers (2024-02-13T12:59:20Z) - Can AI Assistants Know What They Don't Know? [79.6178700946602]
An AI assistant's refusal to answer questions it does not know is a crucial method for reducing hallucinations and making the assistant truthful.
We construct a model-specific "I don't know" (Idk) dataset for an assistant, which contains its known and unknown questions.
After alignment with Idk datasets, the assistant can refuse to answer most its unknown questions.
arXiv Detail & Related papers (2024-01-24T07:34:55Z) - Experiential Co-Learning of Software-Developing Agents [83.34027623428096]
Large language models (LLMs) have brought significant changes to various domains, especially in software development.
We introduce Experiential Co-Learning, a novel LLM-agent learning framework.
Experiments demonstrate that the framework enables agents to tackle unseen software-developing tasks more effectively.
arXiv Detail & Related papers (2023-12-28T13:50:42Z) - Developer Experiences with a Contextualized AI Coding Assistant:
Usability, Expectations, and Outcomes [11.520721038793285]
This study focuses on the initial experiences of 62 participants who used a contextualized coding AI assistant -- named StackSpot AI -- in a controlled setting.
Assistants' use resulted in significant time savings, easier access to documentation, and the generation of accurate codes for internal APIs.
challenges associated with the knowledge sources necessary to make the coding assistant access more contextual information as well as variable responses and limitations in handling complex codes were observed.
arXiv Detail & Related papers (2023-11-30T10:52:28Z) - HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI
Assistants in the Real World [48.90399899928823]
This work is part of a broader research effort to develop intelligent agents that can interactively guide humans through performing tasks in the physical world.
We introduce HoloAssist, a large-scale egocentric human interaction dataset.
We present key insights into how human assistants correct mistakes, intervene in the task completion procedure, and ground their instructions to the environment.
arXiv Detail & Related papers (2023-09-29T07:17:43Z) - The Future of AI-Assisted Writing [0.0]
We conduct a comparative user-study between such tools from an information retrieval lens: pull and push.
Our findings show that users welcome seamless assistance of AI in their writing.
Users also enjoyed the collaboration with AI-assisted writing tools and did not feel a lack of ownership.
arXiv Detail & Related papers (2023-06-29T02:46:45Z) - AI for Low-Code for AI [8.379047663193422]
LowCoder is the first low-code tool for developing AI pipelines that supports both a visual programming interface and an AI-powered natural language interface.
We task 20 developers with varying levels of AI expertise with implementing four ML pipelines using LowCoder.
We find that LowCoder is especially useful for (i) Discoverability: using LowCoder_NL, participants discovered new operators in 75% of the tasks.
arXiv Detail & Related papers (2023-05-31T16:44:03Z) - A Large-Scale Survey on the Usability of AI Programming Assistants:
Successes and Challenges [23.467373994306524]
In practice, developers do not accept AI programming assistants' initial suggestions at a high frequency.
To understand developers' practices while using these tools, we administered a survey to a large population of developers.
We found that developers are most motivated to use AI programming assistants because they help developers reduce key-strokes, finish programming tasks quickly, and recall syntax.
We also found the most important reasons why developers do not use these tools are because these tools do not output code that addresses certain functional or non-functional requirements.
arXiv Detail & Related papers (2023-03-30T03:21:53Z) - Effidit: Your AI Writing Assistant [60.588370965898534]
Effidit is a digital writing assistant that facilitates users to write higher-quality text more efficiently by using artificial intelligence (AI) technologies.
In Effidit, we significantly expand the capacities of a writing assistant by providing functions in five categories: text completion, error checking, text polishing, keywords to sentences (K2S), and cloud input methods (cloud IME)
arXiv Detail & Related papers (2022-08-03T02:24:45Z) - The MineRL BASALT Competition on Learning from Human Feedback [58.17897225617566]
The MineRL BASALT competition aims to spur forward research on this important class of techniques.
We design a suite of four tasks in Minecraft for which we expect it will be hard to write down hardcoded reward functions.
We provide a dataset of human demonstrations on each of the four tasks, as well as an imitation learning baseline.
arXiv Detail & Related papers (2021-07-05T12:18:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.