Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wild

• URL: http://arxiv.org/abs/2607.03316v1
• Date: Fri, 03 Jul 2026 13:35:09 GMT
• Title: Is Agentic Code Review Helpful? Mining Developers' Feedback to CodeRabbit Reviews in the Wild
• Abstract summary: Agentic code review, where autonomous agents provide code review comments on pull requests, is increasingly integrated into development.<n>We present an empirical study of agentic code reviews using CodeRabbit as a case study.
• Score: 4.433169165028139
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic code review, where autonomous agents provide code review comments on pull requests, is increasingly integrated into development workflows, yet there is limited empirical evidence on how developers respond to such comments in practice. In this paper, we present an empirical study of agentic code reviews using CodeRabbit as a case study. Through an empirical study of 31,073 pairs of code reviews and developer feedback from 10,191 pull requests across 239 GitHub repositories, our results show that agentic reviews receive mixed reception: 36.4% were accepted and 7.3% triggered discussion, while 56.3% were rejected. Rejections were primarily associated with invalid suggestions that were false positives, redundant, or out of scope, as well as misalignment with developer intent and coding practices. We further found that agentic reviews tend to focus more on functional concerns than evolvability-related comments, yet they were more likely to be invalid. To improve effectiveness in review practices, we explored various LLM-based approaches for predicting review rejection. We found that lightweight learning-based methods achieve up to 76% F1 score, suggesting learnable patterns exist between code reviews and their corresponding feedback. Our results highlight the current state of CodeRabbit's agentic code reviews, showing opportunity gaps for improvement, as well as shortcomings hindering its effectiveness.

Related papers

• "Go Home Copilot, You're Drunk": Understanding Developer Responses to Agent-Generated Code Review Comments [11.53008696469705]
  We present the first large-scale empirical study on the resolution of agent-generated code review comments.<n>We analyze $54,791 comments generated by five widely used coding agents across $342$ Python repositories on GitHub.<n>Our results show that resolution rate varies considerably across agents, with Copilot accounting for the majority of resolved comments.
  arXiv  Detail & Related papers  (2026-07-24T05:51:48Z)
• SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review [47.72392142544407]
  Coding agents increasingly generate pull requests (PRs) for real-world software issues.<n>One-shot PR generation remains open-loop: the PR is proposed without systematic review, diagnosis, or revision.<n>We introduce textbfSWE-Review, a framework for closing this loop with agentic code review.
  arXiv  Detail & Related papers  (2026-07-07T09:37:45Z)
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues [54.98461672730087]
  We introduce ReproRepo, a scalable framework for evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers.<n>We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations.<n>Our results show that LLM agents, even without executing code, can identify many real-world problems from paper-repository pairs.
  arXiv  Detail & Related papers  (2026-06-16T17:58:05Z)
• What to Cut? Predicting Unnecessary Methods in Agentic Code Generation [0.0]
  We propose a prediction model that identifies functions likely to be deleted during PR review.<n>Our results show that functions deleted for different reasons exhibit distinct characteristics.<n>These findings suggest that predictive approaches can help reviewers prioritize their efforts on essential code.
  arXiv  Detail & Related papers  (2026-02-19T05:29:32Z)
• Understanding Dominant Themes in Reviewing Agentic AI-authored Code [6.183483850365225]
  We analyze 19,450 inline review comments spanning 3,177 agent-authored PRs from real-world GitHub repositories.<n>We find that while AI agents can accelerate code production, there remain gaps requiring targeted human review oversight.
  arXiv  Detail & Related papers  (2026-01-27T07:21:09Z)
• RovoDev Code Reviewer: A Large-Scale Online Evaluation of LLM-based Code Review Automation at Atlassian [6.019311657369222]
  Large Language Models (LLMs)-powered code review automation has the potential to transform seamless code review.<n>This paper aims at answering the practical question: how can we design a review-guided, context-aware, quality-checked code review comment generation without fine-tuning?<n>We present RovoDev Code Reviewer, an enterprise-grade LLM-based code review automation tool designed and deployed at scale within Atlassian's development ecosystem with integration into Atlassian's Bitbucket.
  arXiv  Detail & Related papers  (2026-01-03T09:27:56Z)
• Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 [115.86204862475864]
  Review Feedback Agent provides automated feedback on vague comments, content misunderstandings, and unprofessional remarks to reviewers.<n>It was implemented at ICLR 2025 as a large randomized control study.<n> 27% of reviewers who received feedback updated their reviews, and over 12,000 feedback suggestions from the agent were incorporated by those reviewers.
  arXiv  Detail & Related papers  (2025-04-13T22:01:25Z)
• Understanding Code Understandability Improvements in Code Reviews [79.16476505761582]
  We analyzed 2,401 code review comments from Java open-source projects on GitHub. 83.9% of suggestions for improvement were accepted and integrated, with fewer than 1% later reverted.
  arXiv  Detail & Related papers  (2024-10-29T12:21:23Z)
• Towards debiasing code review support [1.188383832081829]
  This paper explores harmful cases caused by cognitive biases during code review. In particular, we design prototypes covering confirmation bias and decision fatigue. We show that some techniques could be implemented in existing code review tools.
  arXiv  Detail & Related papers  (2024-07-01T15:58:14Z)
• Factoring Expertise, Workload, and Turnover into Code Review Recommendation [4.492444446637857]
  We show that code review natural spreads knowledge thereby reducing the files at risk to turnover. We develop novel recommenders to understand their impact on the level of expertise during review. We are able to globally increase expertise during reviews, +3%, reduce workload concentration, -12%, and reduce the files at risk, -28%.
  arXiv  Detail & Related papers  (2023-12-28T18:58:06Z)
• Code Review in the Classroom [57.300604527924015]
  Young developers in a classroom setting provide a clear picture of the potential favourable and problematic areas of the code review process. Their feedback suggests that the process has been well received with some points to better the process. This paper can be used as guidelines to perform code reviews in the classroom.
  arXiv  Detail & Related papers  (2020-04-19T06:07:45Z)
• Automating App Review Response Generation [67.58267006314415]
  We propose a novel approach RRGen that automatically generates review responses by learning knowledge relations between reviews and their responses. Experiments on 58 apps and 309,246 review-response pairs highlight that RRGen outperforms the baselines by at least 67.4% in terms of BLEU-4.
  arXiv  Detail & Related papers  (2020-02-10T05:23:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.