"Go Home Copilot, You're Drunk": Understanding Developer Responses to Agent-Generated Code Review Comments

• URL: http://arxiv.org/abs/2607.21997v2
• Date: Wed, 29 Jul 2026 20:44:35 GMT
• Title: "Go Home Copilot, You're Drunk": Understanding Developer Responses to Agent-Generated Code Review Comments
• Abstract summary: We present the first large-scale empirical study on the resolution of agent-generated code review comments.<n>We analyze $54,791 comments generated by five widely used coding agents across $342$ Python repositories on GitHub.<n>Our results show that resolution rate varies considerably across agents, with Copilot accounting for the majority of resolved comments.
• Score: 11.53008696469705
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Code review is a critical quality assurance practice in software engineering development, and AI coding agents are increasingly generating review comments on pull requests. However, little is known about how developers actually respond to such agent-generated feedback. In this paper, we present the first large-scale empirical study on the resolution of agent-generated code review comments. We analyze $54{,}791$ comments generated by five widely used coding agents (i.e., Copilot, Cursor, Codex, Devin, and Claude) across $342$ Python repositories on GitHub. We examine (1) resolution rates across agents and comment types, (2) the role of developer experience, and (3) characteristics that influence comment usefulness. Our results show that resolution rate varies considerably across agents, with Copilot accounting for the majority of resolved comments (72.9\%). Core developers resolve the majority of agent-generated feedback, particularly for \textit{design} and \textit{evolvability}-related comments, while peripheral developers are more involved in resolving \textit{functional defect} comments. Through open card sorting of 470 unresolved comment discussions, we identify \textit{ten} discussion patterns explaining why comments remain unresolved, with \textit{incorrect suggestions} and \textit{intentional design decisions} being the most prevalent. Finally, our analysis reveals that the presence of an inline \textit{code suggestion} is the strongest predictor of comment resolution, while lengthy and complex comments are less likely to be acted upon. Our findings provide insights for improving AI-generated code review feedback and its integration into development workflows.

Related papers

• (Im)Paired Programming: Coding Agents Improve Productivity but Harm Understanding [59.19634073105828]
  We have 54 students create a website with one of two AI systems: an agent that edits user code.<n>We test understanding via comprehension questions and a task where users extend their code without agents.<n>Despite self-reported weaker understanding, users still prefer coding agents because they are quick and easy to use.
  arXiv  Detail & Related papers  (2026-07-29T01:23:04Z)
• Code Review is a Conversation: Toward Conversational AI Review Assistants [7.335667281862373]
  We argue for conversational AI review assistants that participate in code review as interactive partners rather than static comment generators.<n>Such assistants should identify when conversation is needed, ask grounded questions, respond to developer explanations, summarize unresolved issues, help capture rationale, and know when to abstain or escalate to human reviewers.
  arXiv  Detail & Related papers  (2026-07-24T08:44:05Z)
• SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review [47.72392142544407]
  Coding agents increasingly generate pull requests (PRs) for real-world software issues.<n>One-shot PR generation remains open-loop: the PR is proposed without systematic review, diagnosis, or revision.<n>We introduce textbfSWE-Review, a framework for closing this loop with agentic code review.
  arXiv  Detail & Related papers  (2026-07-07T09:37:45Z)
• Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wild [4.433169165028139]
  Agentic code review, where autonomous agents provide code review comments on pull requests, is increasingly integrated into development.<n>We present an empirical study of agentic code reviews using CodeRabbit as a case study.
  arXiv  Detail & Related papers  (2026-07-03T13:35:09Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• What Types of Code Review Comments Do Developers Most Frequently Resolve? [10.277847378685161]
  Large language model (LLM)-powered code review automation tools have been introduced to generate code review comments.<n>This paper investigates the types of review comments written by humans and LLMs, and the types of generated comments that are most frequently resolved by developers.
  arXiv  Detail & Related papers  (2025-10-06T23:32:26Z)
• Does AI Code Review Lead to Code Changes? A Case Study of GitHub Actions [21.347559936084807]
  AI-based code review tools automatically review and comment on pull requests to improve code quality.<n>We present a large-scale empirical study of 16 popular AI-based code review actions for GitHub.<n>We investigate how these tools are adopted and configured, whether their comments lead to code changes, and which factors influence their effectiveness.
  arXiv  Detail & Related papers  (2025-08-26T07:55:23Z)
• RedCode: Risky Code Execution and Generation Benchmark for Code Agents [50.81206098588923]
  RedCode is a benchmark for risky code execution and generation. RedCode-Exec provides challenging prompts that could lead to risky code execution. RedCode-Gen provides 160 prompts with function signatures and docstrings as input to assess whether code agents will follow instructions.
  arXiv  Detail & Related papers  (2024-11-12T13:30:06Z)
• Understanding Code Understandability Improvements in Code Reviews [79.16476505761582]
  We analyzed 2,401 code review comments from Java open-source projects on GitHub. 83.9% of suggestions for improvement were accepted and integrated, with fewer than 1% later reverted.
  arXiv  Detail & Related papers  (2024-10-29T12:21:23Z)
• Towards debiasing code review support [1.188383832081829]
  This paper explores harmful cases caused by cognitive biases during code review. In particular, we design prototypes covering confirmation bias and decision fatigue. We show that some techniques could be implemented in existing code review tools.
  arXiv  Detail & Related papers  (2024-07-01T15:58:14Z)
• Are your comments outdated? Towards automatically detecting code-comment consistency [3.204922482708544]
  Outdated comments are dangerous and harmful and may mislead subsequent developers. We propose a learning-based method, called CoCC, to detect the consistency between code and comment. Experiment results show that CoCC can effectively detect outdated comments with precision over 90%.
  arXiv  Detail & Related papers  (2024-03-01T03:30:13Z)
• Exploring the Advances in Identifying Useful Code Review Comments [0.0]
  This paper reflects the evolution of research on the usefulness of code review comments. It examines papers that define the usefulness of code review comments, mine and annotate datasets, study developers' perceptions, analyze factors from different aspects, and use machine learning classifiers to automatically predict the usefulness of code review comments.
  arXiv  Detail & Related papers  (2023-07-03T00:41:20Z)
• Can We Automate Scientific Reviewing? [89.50052670307434]
  We discuss the possibility of using state-of-the-art natural language processing (NLP) models to generate first-pass peer reviews for scientific papers. We collect a dataset of papers in the machine learning domain, annotate them with different aspects of content covered in each review, and train targeted summarization models that take in papers to generate reviews. Comprehensive experimental results show that system-generated reviews tend to touch upon more aspects of the paper than human-written reviews.
  arXiv  Detail & Related papers  (2021-01-30T07:16:53Z)
• Deep Just-In-Time Inconsistency Detection Between Comments and Source Code [51.00904399653609]
  In this paper, we aim to detect whether a comment becomes inconsistent as a result of changes to the corresponding body of code. We develop a deep-learning approach that learns to correlate a comment with code changes. We show the usefulness of our approach by combining it with a comment update model to build a more comprehensive automatic comment maintenance system.
  arXiv  Detail & Related papers  (2020-10-04T16:49:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.