Cracking CodeWhisperer: Analyzing Developers' Interactions and Patterns During Programming Tasks

• URL: http://arxiv.org/abs/2510.11516v1
• Date: Mon, 13 Oct 2025 15:22:12 GMT
• Title: Cracking CodeWhisperer: Analyzing Developers' Interactions and Patterns During Programming Tasks
• Abstract summary: We investigate how developers engage with Amazon's CodeWhisperer, an LLM-based code-generation tool.<n>We conducted two user studies with two groups of 10 participants each.<n>We identify four behavioral patterns: 1) incremental code refinement, 2) explicit instruction using natural language comments, 3) baseline structuring with model suggestions, and 4) integrative use with external sources.
• Score: 4.321456509547389
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The use of AI code-generation tools is becoming increasingly common, making it important to understand how software developers are adopting these tools. In this study, we investigate how developers engage with Amazon's CodeWhisperer, an LLM-based code-generation tool. We conducted two user studies with two groups of 10 participants each, interacting with CodeWhisperer - the first to understand which interactions were critical to capture and the second to collect low-level interaction data using a custom telemetry plugin. Our mixed-methods analysis identified four behavioral patterns: 1) incremental code refinement, 2) explicit instruction using natural language comments, 3) baseline structuring with model suggestions, and 4) integrative use with external sources. We provide a comprehensive analysis of these patterns .

Related papers

• Lost in the Flow with Code Talkers: Unveiling the Instruction-Tuning Tax of Large Language Models in Code Tasks [6.645779775557948]
  We conduct the first empirical study that uncovers a key trade-off caused by instruction tuning across programming modes.<n>Our results show that instruction tuning is not a free lunch: although instruction-tuned models are more capable of following instructions, these gains often come at the cost of weaker infilling performance.
  arXiv  Detail & Related papers  (2026-06-07T15:24:30Z)
• Decoding Human-LLM Collaboration in Coding: An Empirical Study of Multi-Turn Conversations in the Wild [15.241064679369407]
  We conduct an empirical analysis on human-LLM coding collaboration using LMSYS-Chat-1M and WildChat datasets.<n>We find that task types shape interaction patterns, with code quality optimization favoring linear patterns, design-driven tasks leaning toward tree structures, and queries preferring star patterns.<n>We believe this work broadens understanding of human-LLM synergies and supports more effective AI-assisted development.
  arXiv  Detail & Related papers  (2025-12-11T10:14:42Z)
• From Code Foundation Models to Agents and Applications: A Practical Guide to Code Intelligence [150.3696990310269]
  Large language models (LLMs) have transformed automated software development by enabling direct translation of natural language descriptions into functional code.<n>We provide a comprehensive synthesis and practical guide (a series of analytic and probing experiments) about code LLMs.<n>We analyze the code capability of the general LLMs (GPT-4, Claude, LLaMA) and code-specialized LLMs (StarCoder, Code LLaMA, DeepSeek-Coder, and QwenCoder)
  arXiv  Detail & Related papers  (2025-11-23T17:09:34Z)
• LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering [90.84806758077536]
  We introduce textbfLoCoBench-Agent, a comprehensive evaluation framework specifically designed to assess large language models (LLMs) agents in realistic, long-context software engineering.<n>Our framework extends LoCoBench's 8,000 scenarios into interactive agent environments, enabling systematic evaluation of multi-turn conversations.<n>Our framework provides agents with 8 specialized tools (file operations, search, code analysis) and evaluates them across context lengths ranging from 10K to 1M tokens.
  arXiv  Detail & Related papers  (2025-11-17T23:57:24Z)
• A Survey of Vibe Coding with Large Language Models [93.88284590533242]
  "Vibe Coding" is a development methodology where developers validate AI-generated implementations through outcome observation.<n>Despite its transformative potential, the effectiveness of this emergent paradigm remains under-explored.<n>This survey provides the first comprehensive and systematic review of Vibe Coding with large language models.
  arXiv  Detail & Related papers  (2025-10-14T11:26:56Z)
• AI-Guided Exploration of Large-Scale Codebases [0.0]
  Large language models (LLMs) offer new opportunities to enhance code exploration.<n>Recent advancements in large language models (LLMs) offer new opportunities to enhance code exploration.<n>This work introduces a hybrid approach that integrates reverse engineering with LLM-guided, intent-aware visual exploration.
  arXiv  Detail & Related papers  (2025-08-07T19:15:37Z)
• Tool-Star: Empowering LLM-Brained Multi-Tool Reasoner via Reinforcement Learning [63.31585771716123]
  Large language models (LLMs) have shown remarkable reasoning capabilities via large-scale reinforcement learning (RL)<n>We introduce Tool-Star, an RL-based framework designed to empower LLMs to autonomously invoke multiple external tools during stepwise reasoning.<n>Tool-Star integrates six types of tools and incorporates systematic designs in both data synthesis and training.
  arXiv  Detail & Related papers  (2025-05-22T09:00:19Z)
• FamilyTool: A Multi-hop Personalized Tool Use Benchmark [93.80355496575281]
  FamilyTool is a benchmark grounded in a family-based knowledge graph (KG) that simulates personalized, multi-hop tool use scenarios.<n> Experiments reveal significant performance gaps in state-of-the-art Large Language Models (LLMs)<n>FamilyTool serves as a critical resource for evaluating and advancing LLM agents' reasoning, adaptability, and scalability in complex, dynamic environments.
  arXiv  Detail & Related papers  (2025-04-09T10:42:36Z)
• Training of Scaffolded Language Models with Language Supervision: A Survey [62.59629932720519]
  This survey organizes the literature on the design and optimization of emerging structures around post-trained LMs.<n>We refer to this overarching structure as scaffolded LMs and focus on LMs that are integrated into multi-step processes with tools.
  arXiv  Detail & Related papers  (2024-10-21T18:06:25Z)
• LLM With Tools: A Survey [0.0]
  This paper delves into the methodology,challenges, and developments in the realm of teaching LLMs to use external tools. We introduce a standardized paradigm for tool integration guided by a series of functions that map user instructions to actionable plans. Our exploration reveals the various challenges encountered, such as tool invocation timing, selection accuracy, and the need for robust reasoning processes.
  arXiv  Detail & Related papers  (2024-09-24T14:08:11Z)
• What You Need is What You Get: Theory of Mind for an LLM-Based Code Understanding Assistant [0.0]
  A growing number of tools have used Large Language Models (LLMs) to support developers' code understanding. In this study, we designed an LLM-based conversational assistant that provides a personalized interaction based on inferred user mental state. Our results provide insights for researchers and tool builders who want to create or improve LLM-based conversational assistants to support novices in code understanding.
  arXiv  Detail & Related papers  (2024-08-08T14:08:15Z)
• Exploring Interaction Patterns for Debugging: Enhancing Conversational Capabilities of AI-assistants [18.53732314023887]
  Large Language Models (LLMs) enable programmers to obtain natural language explanations for various software development tasks. LLMs often leap to action without sufficient context, giving rise to implicit assumptions and inaccurate responses. In this paper, we draw inspiration from interaction patterns and conversation analysis -- to design Robin, an enhanced conversational AI-assistant for debug.
  arXiv  Detail & Related papers  (2024-02-09T07:44:27Z)
• MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback [78.60644407028022]
  We introduce MINT, a benchmark that evaluates large language models' ability to solve tasks with multi-turn interactions. LLMs generally benefit from tools and language feedback, with performance gains of 1-8% for each turn of tool use. LLMs evaluated, supervised instruction-finetuning (SIFT) and reinforcement learning from human feedback (RLHF) generally hurt multi-turn capabilities.
  arXiv  Detail & Related papers  (2023-09-19T15:25:42Z)
• Using an LLM to Help With Code Understanding [13.53616539787915]
  Large language models (LLMs) are revolutionizing the process of writing code. Our plugin queries OpenAI's GPT-3.5-turbo model with four high-level requests without the user having to write explicit prompts. We evaluate this system in a user study with 32 participants, which confirms that using our plugin can aid task completion more than web search.
  arXiv  Detail & Related papers  (2023-07-17T00:49:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.