Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
- URL: http://arxiv.org/abs/2210.14306v5
- Date: Mon, 22 Apr 2024 04:00:10 GMT
- Title: Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
- Authors: Hussein Mozannar, Gagan Bansal, Adam Fourney, Eric Horvitz,
- Abstract summary: We studied GitHub Copilot, a code-recommendation system used by millions of programmers daily.
We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot.
Our insights reveal how programmers interact with Copilot and motivate new interface designs and metrics.
- Score: 28.254978977288868
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Code-recommendation systems, such as Copilot and CodeWhisperer, have the potential to improve programmer productivity by suggesting and auto-completing code. However, to fully realize their potential, we must understand how programmers interact with these systems and identify ways to improve that interaction. To seek insights about human-AI collaboration with code recommendations systems, we studied GitHub Copilot, a code-recommendation system used by millions of programmers daily. We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot. Our study of 21 programmers, who completed coding tasks and retrospectively labeled their sessions with CUPS, showed that CUPS can help us understand how programmers interact with code-recommendation systems, revealing inefficiencies and time costs. Our insights reveal how programmers interact with Copilot and motivate new interface designs and metrics.
Related papers
- OpenHands: An Open Platform for AI Software Developers as Generalist Agents [109.8507367518992]
We introduce OpenHands, a platform for the development of AI agents that interact with the world in similar ways to a human developer.
We describe how the platform allows for the implementation of new agents, safe interaction with sandboxed environments for code execution, and incorporation of evaluation benchmarks.
arXiv Detail & Related papers (2024-07-23T17:50:43Z) - Automatic Programming: Large Language Models and Beyond [48.34544922560503]
We study concerns around code quality, security and related issues of programmer responsibility.
We discuss how advances in software engineering can enable automatic programming.
We conclude with a forward looking view, focusing on the programming environment of the near future.
arXiv Detail & Related papers (2024-05-03T16:19:24Z) - Rethinking Software Engineering in the Foundation Model Era: From Task-Driven AI Copilots to Goal-Driven AI Pair Programmers [30.996760992473064]
We propose a paradigm shift towards goal-driven AI-powered pair programmers that collaborate with human developers.
We envision AI pair programmers that are goal-driven, human partners, SE-aware, and self-learning.
arXiv Detail & Related papers (2024-04-16T02:10:20Z) - Charting a Path to Efficient Onboarding: The Role of Software
Visualization [49.1574468325115]
The present study aims to explore the familiarity of managers, leaders, and developers with software visualization tools.
This approach incorporated quantitative and qualitative analyses of data collected from practitioners using questionnaires and semi-structured interviews.
arXiv Detail & Related papers (2024-01-17T21:30:45Z) - PwR: Exploring the Role of Representations in Conversational Programming [17.838776812138626]
We introduce Programming with Representations (PwR), an approach that uses representations to convey the system's understanding back to the user in natural language.
We find that representations significantly improve understandability, and instilled a sense of agency among our participants.
arXiv Detail & Related papers (2023-09-18T05:38:23Z) - Collaborative, Code-Proximal Dynamic Software Visualization within Code
Editors [55.57032418885258]
This paper introduces the design and proof-of-concept implementation for a software visualization approach that can be embedded into code editors.
Our contribution differs from related work in that we use dynamic analysis of a software system's runtime behavior.
Our visualization approach enhances common remote pair programming tools and is collaboratively usable by employing shared code cities.
arXiv Detail & Related papers (2023-08-30T06:35:40Z) - When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming [28.254978977288868]
We pursue mechanisms for leveraging signals about programmers' acceptance and rejection of code suggestions to guide recommendations.
We introduce a utility-theoretic framework to drive decisions about suggestions to display versus withhold.
conditional suggestion display from human feedback relies on a cascade of models that provide the likelihood that recommended code will be accepted.
arXiv Detail & Related papers (2023-06-08T04:24:24Z) - "It's Weird That it Knows What I Want": Usability and Interactions with
Copilot for Novice Programmers [5.317693153442043]
We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical programming assignment.
We explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students.
arXiv Detail & Related papers (2023-04-05T15:07:50Z) - Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science.
Standard approaches require instructors to manually grade student-implemented interactive programs.
Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
arXiv Detail & Related papers (2022-11-16T10:00:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.