Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming

• URL: http://arxiv.org/abs/2210.14306v5
• Date: Mon, 22 Apr 2024 04:00:10 GMT
• Title: Reading Between the Lines: Modeling User Behavior and Costs in AI-Assisted Programming
• Abstract summary: We studied GitHub Copilot, a code-recommendation system used by millions of programmers daily. We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot. Our insights reveal how programmers interact with Copilot and motivate new interface designs and metrics.
• Score: 28.254978977288868
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Code-recommendation systems, such as Copilot and CodeWhisperer, have the potential to improve programmer productivity by suggesting and auto-completing code. However, to fully realize their potential, we must understand how programmers interact with these systems and identify ways to improve that interaction. To seek insights about human-AI collaboration with code recommendations systems, we studied GitHub Copilot, a code-recommendation system used by millions of programmers daily. We developed CUPS, a taxonomy of common programmer activities when interacting with Copilot. Our study of 21 programmers, who completed coding tasks and retrospectively labeled their sessions with CUPS, showed that CUPS can help us understand how programmers interact with code-recommendation systems, revealing inefficiencies and time costs. Our insights reveal how programmers interact with Copilot and motivate new interface designs and metrics.

Related papers

• Code as Agent Harness [107.31925305395957]
  In emerging agentic systems, code is no longer only a target output.<n>Code increasingly serves as an operational substrate for agent reasoning, acting, environment modeling, and execution-based verification.<n>This survey provides a unified roadmap toward executable, verifiable, and stateful AI agent systems.
  arXiv  Detail & Related papers  (2026-05-18T17:59:03Z)
• From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
  Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
  arXiv  Detail & Related papers  (2025-11-23T17:09:34Z)
• Computer-Use Agents as Judges for Generative User Interface [142.75272102498806]
  ComputerUse Agents (CUA) are becoming increasingly capable of autonomously operating digital environments through Graphical User Interfaces (GUI)<n>Most GUI remain designed primarily for humans to adopt human-oriented behaviors that are unnecessary for efficient task execution.<n>This raises a fundamental question: Can CUA as judges to assist Coder for automatic GUI design?
  arXiv  Detail & Related papers  (2025-11-19T16:00:02Z)
• Evolution of Programmers' Trust in Generative AI Programming Assistants [2.006647079907382]
  This study aims to understand programmers' evolution of trust after immediate (one hour) and extended (10 days) use of GitHub Copilot.<n>After completing a project with Copilot, students felt that Copilot requires a competent programmer to complete some tasks manually.<n>Students mentioned that seeing Copilot's correctness, understanding how Copilot uses context from the code base, and learning some basics of natural language processing contributed to their elevated trust.
  arXiv  Detail & Related papers  (2025-09-16T17:06:47Z)
• A Human Centric Requirements Engineering Framework for Assessing Github Copilot Output [0.0]
  GitHub Copilot introduces new challenges in how these software tools address human needs.<n>I analyzed GitHub Copilot's interaction with users through its chat interface.<n>I established a human-centered requirements framework with clear metrics to evaluate these qualities.
  arXiv  Detail & Related papers  (2025-08-05T21:33:23Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [66.1850490474361]
  We conduct the first academic study to explore developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants, GitHub Copilot and OpenHands.<n>Our results show agents have the potential to assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• OpenHands: An Open Platform for AI Software Developers as Generalist Agents [109.8507367518992]
  We introduce OpenHands, a platform for the development of AI agents that interact with the world in similar ways to a human developer. We describe how the platform allows for the implementation of new agents, safe interaction with sandboxed environments for code execution, and incorporation of evaluation benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T17:50:43Z)
• Automatic Programming: Large Language Models and Beyond [48.34544922560503]
  We study concerns around code quality, security and related issues of programmer responsibility. We discuss how advances in software engineering can enable automatic programming. We conclude with a forward looking view, focusing on the programming environment of the near future.
  arXiv  Detail & Related papers  (2024-05-03T16:19:24Z)
• Rethinking Software Engineering in the Foundation Model Era: From Task-Driven AI Copilots to Goal-Driven AI Pair Programmers [30.996760992473064]
  We propose a paradigm shift towards goal-driven AI-powered pair programmers that collaborate with human developers. We envision AI pair programmers that are goal-driven, human partners, SE-aware, and self-learning.
  arXiv  Detail & Related papers  (2024-04-16T02:10:20Z)
• Charting a Path to Efficient Onboarding: The Role of Software Visualization [49.1574468325115]
  The present study aims to explore the familiarity of managers, leaders, and developers with software visualization tools. This approach incorporated quantitative and qualitative analyses of data collected from practitioners using questionnaires and semi-structured interviews.
  arXiv  Detail & Related papers  (2024-01-17T21:30:45Z)
• PwR: Exploring the Role of Representations in Conversational Programming [17.838776812138626]
  We introduce Programming with Representations (PwR), an approach that uses representations to convey the system's understanding back to the user in natural language. We find that representations significantly improve understandability, and instilled a sense of agency among our participants.
  arXiv  Detail & Related papers  (2023-09-18T05:38:23Z)
• Collaborative, Code-Proximal Dynamic Software Visualization within Code Editors [55.57032418885258]
  This paper introduces the design and proof-of-concept implementation for a software visualization approach that can be embedded into code editors. Our contribution differs from related work in that we use dynamic analysis of a software system's runtime behavior. Our visualization approach enhances common remote pair programming tools and is collaboratively usable by employing shared code cities.
  arXiv  Detail & Related papers  (2023-08-30T06:35:40Z)
• When to Show a Suggestion? Integrating Human Feedback in AI-Assisted Programming [28.254978977288868]
  We pursue mechanisms for leveraging signals about programmers' acceptance and rejection of code suggestions to guide recommendations. We introduce a utility-theoretic framework to drive decisions about suggestions to display versus withhold. conditional suggestion display from human feedback relies on a cascade of models that provide the likelihood that recommended code will be accepted.
  arXiv  Detail & Related papers  (2023-06-08T04:24:24Z)
• "It's Weird That it Knows What I Want": Usability and Interactions with Copilot for Novice Programmers [5.317693153442043]
  We present the first study that observes students at the introductory level using one such code auto-generating tool, Github Copilot, on a typical programming assignment. We explore student perceptions of the benefits and pitfalls of this technology for learning, present new observed interaction patterns, and discuss cognitive and metacognitive difficulties faced by students.
  arXiv  Detail & Related papers  (2023-04-05T15:07:50Z)
• Giving Feedback on Interactive Student Programs with Meta-Exploration [74.5597783609281]
  Developing interactive software, such as websites or games, is a particularly engaging way to learn computer science. Standard approaches require instructors to manually grade student-implemented interactive programs. Online platforms that serve millions, like Code.org, are unable to provide any feedback on assignments for implementing interactive programs.
  arXiv  Detail & Related papers  (2022-11-16T10:00:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.