Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions

• URL: http://arxiv.org/abs/2604.00436v1
• Date: Wed, 01 Apr 2026 03:30:53 GMT
• Title: Programming by Chat: A Large-Scale Behavioral Analysis of 11,579 Real-World AI-Assisted IDE Sessions
• Abstract summary: We analyze 74,998 developer messages from 11,579 chat sessions across 1,300 repositories and 899 developers using Cursor and GitHub Copilot.<n>Our findings reveal three shifts in how programming work is organized: developers iteratively refining outputs rather than specifying complete tasks upfront; developers redistribute cognitive work to AI, delegating diagnosis, comprehension, and validation rather than engaging with code and outputs directly; and developers actively manage the collaboration, externalizing plans into persistent artifacts, and negotiating AI autonomy through context injection and behavioral constraints.
• Score: 16.76348030799203
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: IDE-integrated AI coding assistants, which operate conversationally within developers' working codebases with access to project context and multi-file editing, are rapidly reshaping software development. However, empirical investigation of this shift remains limited: existing studies largely rely on small-scale, controlled settings or analyze general-purpose chatbots rather than codebase-aware IDE workflows. We present, to the best of our knowledge, the first large-scale study of real-world conversational programming in IDE-native settings, analyzing 74,998 developer messages from 11,579 chat sessions across 1,300 repositories and 899 developers using Cursor and GitHub Copilot. These chats were committed to public repositories as part of routine development, capturing in-the-wild behavior. Our findings reveal three shifts in how programming work is organized: conversational programming operates as progressive specification, with developers iteratively refining outputs rather than specifying complete tasks upfront; developers redistribute cognitive work to AI, delegating diagnosis, comprehension, and validation rather than engaging with code and outputs directly; and developers actively manage the collaboration, externalizing plans into persistent artifacts, and negotiating AI autonomy through context injection and behavioral constraints. These results provide foundational empirical insights into AI-assisted development and offer implications for the design of future programming environments.

Related papers

• How Developers Discuss Generative AI: A Longitudinal Study of the Visual Studio Code Community [0.0]
  Generative AI tools such as GitHub Copilot, ChatGPT, and coding agents have rapidly become part of everyday software development.<n>This paper presents a longitudinal analysis of generative-AI-related discussions in the Visual Studio Code (VS Code) GitHub repository.
  arXiv  Detail & Related papers  (2026-09-04T03:16:08Z)
• From Conversation to Contribution: Characterizing Coding Agent in Open-Source Software [15.414964008939434]
  We study how developers' chat-based interactions with AI relate to subsequent open-source development and collaboration.<n>We find heavier AI use in smaller, less mature, and less collaborative repositories.<n>While most developers are willing to share their chat, concerns remain around appearing incompetent.
  arXiv  Detail & Related papers  (2026-07-06T22:44:45Z)
• From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI [74.72113315839412]
  Large Language Models (LLMs) are undergoing a fundamental transformation into integrated AI systems capable of reasoning, memory, and self-improvement.<n>We conceptualize this transition as a shift from episodic to Digital Colleague: from conversational answers to persistent work.
  arXiv  Detail & Related papers  (2026-06-12T14:33:55Z)
• RECAP: An End-to-End Platform for Capturing, Replaying, and Analyzing AI-Assisted Programming Interactions [21.75814643077238]
  We present RECAP (Replay and Examine Captured AI Programming), an open-source platform that passively records AI chat sessions and fine-grained code edits inside VS Code without disrupting the developer's workflow.<n>We demonstrate how the platform's linked data and replay capabilities enable analyses of developer-AI interaction patterns that no single data source could support.
  arXiv  Detail & Related papers  (2026-05-01T21:20:38Z)
• "Can you feel the vibes?": An exploration of novice programmer engagement with vibe coding [42.82674998306379]
  "vibe coding" refers to creating software via natural language prompts rather than direct code authorship.<n>This paper reports on a one-day educational hackathon investigating how novice programmers and mixed-experience teams engage with vibe coding.
  arXiv  Detail & Related papers  (2025-12-02T13:32:23Z)
• From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
  Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
  arXiv  Detail & Related papers  (2025-11-23T17:09:34Z)
• A Survey of Vibe Coding with Large Language Models [93.88284590533242]
  "Vibe Coding" is a development methodology where developers validate AI-generated implementations through outcome observation.<n>Despite its transformative potential, the effectiveness of this emergent paradigm remains under-explored.<n>This survey provides the first comprehensive and systematic review of Vibe Coding with large language models.
  arXiv  Detail & Related papers  (2025-10-14T11:26:56Z)
• Prompting in Practice: Investigating Software Developers' Use of Generative AI Tools [17.926187565860232]
  The integration of generative artificial intelligence (GenAI) tools has fundamentally transformed software development.<n>This study presents a systematic investigation of how software engineers integrate GenAI tools into their professional practice.<n>We surveyed 91 software engineers, including 72 active GenAI users, to understand AI usage patterns throughout the development process.
  arXiv  Detail & Related papers  (2025-10-07T15:02:22Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [60.04362496037186]
  We present the first controlled study of developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants.<n>Our results show agents can assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• Enhancing Software Development with Context-Aware Conversational Agents: A User Study on Developer Interactions with Chatbots [3.6321891270689055]
  We conducted a user study with 29 developers using a prototype text-based chatbots to investigate preferred functionalities.<n>Our findings reveal strong interest in task automation, version control support, and contextual adaptability.<n>We highlight the importance of deep contextual understanding, historical interaction awareness, and personalized support in CA design.
  arXiv  Detail & Related papers  (2025-05-13T15:08:55Z)
• OpenHands: An Open Platform for AI Software Developers as Generalist Agents [109.8507367518992]
  We introduce OpenHands, a platform for the development of AI agents that interact with the world in similar ways to a human developer.<n>We describe how the platform allows for the implementation of new agents, safe interaction with sandboxed environments for code execution, and incorporation of evaluation benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T17:50:43Z)
• Developer Experiences with a Contextualized AI Coding Assistant: Usability, Expectations, and Outcomes [11.520721038793285]
  This study focuses on the initial experiences of 62 participants who used a contextualized coding AI assistant -- named StackSpot AI -- in a controlled setting. Assistants' use resulted in significant time savings, easier access to documentation, and the generation of accurate codes for internal APIs. challenges associated with the knowledge sources necessary to make the coding assistant access more contextual information as well as variable responses and limitations in handling complex codes were observed.
  arXiv  Detail & Related papers  (2023-11-30T10:52:28Z)
• ChatDev: Communicative Agents for Software Development [84.90400377131962]
  ChatDev is a chat-powered software development framework in which specialized agents are guided in what to communicate. These agents actively contribute to the design, coding, and testing phases through unified language-based communication.
  arXiv  Detail & Related papers  (2023-07-16T02:11:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.