On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories

• URL: http://arxiv.org/abs/2604.24450v1
• Date: Mon, 27 Apr 2026 13:17:13 GMT
• Title: On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories
• Abstract summary: This study investigates the relationship between reviewer-bot feedback and PR outcomes.<n>We analyze 7,416 reviewer-bot comments on 4,532 PRs from the AI_Dev dataset.<n>Our results show that reviewer-bot comments mainly focus on bug fixes, testing, and documentation, are civil in tone, and are prescriptive in nature.
• Score: 2.155083987188202
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autonomous coding agents are reshaping software development by creating pull requests (PRs) on GitHub, referred to as agentic PRs. In parallel, the review process is also becoming autonomous, thereby making reviewer bots key actors in the assessment of these agentic PRs. However, their influence on PR acceptance and resolution remains unclear. This study empirically investigates the relationship between reviewer-bot feedback and PR outcomes by analyzing how Reviewer Bot Feedback Quality (relevance, clarity, conciseness) and Reviewer Bot Activity Volume (comment count) are associated with PR acceptance and resolution time. We analyze 7,416 reviewer-bot comments on 4,532 PRs from the AI_Dev dataset (a dataset that captured AI agents' PRs in GitHub projects). Our results show that reviewer-bot comments mainly focus on bug fixes, testing, and documentation, are civil in tone, and are prescriptive in nature. Reviewer bots generally produce clear and concise feedback, though the semantic relevance of comments to underlying code changes is moderate. We find that higher Reviewer Bot Activity volume is associated with longer PR resolution times and lower average feedback quality, showing that as bots generate more comments on a PR, the average pertinence of that feedback appears to degrade. At the same time, Reviewer Bot Feedback Quality shows no meaningful association with workflow outcomes. Our findings suggest that, in agentic PR workflows, reviewer bots should prioritize targeted high-relevance feedback over generating large numbers of comments.

Related papers

• Predicting Acceptance and Review Effort in Human and Agent Pull Requests [6.709069732781736]
  Pull requests (PRs) are a central mechanism for reviewing and integrating code changes in modern software repositories.<n>This paper studies whether outcomes can be estimated at the time a PR is opened, before reviewer discussion, CI feedback, or merge decisions are available.<n>We construct a leakage-aware prediction pipeline for human- and agent-authored PRs.
  arXiv  Detail & Related papers  (2026-07-13T18:18:27Z)
• SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review [47.72392142544407]
  Coding agents increasingly generate pull requests (PRs) for real-world software issues.<n>One-shot PR generation remains open-loop: the PR is proposed without systematic review, diagnosis, or revision.<n>We introduce textbfSWE-Review, a framework for closing this loop with agentic code review.
  arXiv  Detail & Related papers  (2026-07-07T09:37:45Z)
• Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code [7.362328661065831]
  We study repeat reviewers who collectively reviewed over a seven-month observation period.<n>We observe a population-level shift in approval rate from 30.1% to 36.8%.<n>This shift is experience-driven (persists after controlling for calendar time), agent-specific latency (human PR approval rates decline over the same period), and not explained by PR difficulty (median PR size is flat).<n>The combination of rising approval, declining comment effort, and increasing queue time is most consistent with reflexive habituation under growing workload rather than rational trust calibration alone.
  arXiv  Detail & Related papers  (2026-06-21T23:38:33Z)
• Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review [2.2807488730605296]
  We present a vision for an AI-powered code review workflow combining specialized agents with human-controlled quality.<n>We identify major open challenges for responsible adoption, including reliability, bias, privacy, automation bias, transparency, and evaluation.
  arXiv  Detail & Related papers  (2026-05-17T17:04:21Z)
• These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests [0.0]
  We analyze code review interactions for AI-generated pull requests (PRs) on GitHub using the AIDev dataset.<n>We find that most AI-generated PRs receive no review and, when reviewed, are largely dominated by AI agents rather than humans.
  arXiv  Detail & Related papers  (2026-05-04T06:32:50Z)
• ReviewGrounder: Improving Review Substantiveness with Rubric-Guided, Tool-Integrated Agents [50.27474750319121]
  Review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews.<n>We introduce REVIEWBENCH, a benchmark evaluating review text according to paper-specific rubrics derived from official guidelines, the paper's content, and human-written reviews.<n>We propose REVIEWGROUNDER, a rubric-guided, tool-integrated multi-agent framework that decomposes reviewing into drafting and grounding stages.
  arXiv  Detail & Related papers  (2026-04-15T16:33:04Z)
• Understanding Dominant Themes in Reviewing Agentic AI-authored Code [6.183483850365225]
  We analyze 19,450 inline review comments spanning 3,177 agent-authored PRs from real-world GitHub repositories.<n>We find that while AI agents can accelerate code production, there remain gaps requiring targeted human review oversight.
  arXiv  Detail & Related papers  (2026-01-27T07:21:09Z)
• On Autopilot? An Empirical Study of Human-AI Teaming and Review Practices in Open Source [11.412808537439973]
  We investigated project-level guidelines and developers' interactions with AI-assisted pull requests (PRs)<n>We found that over 67.5% of AI-co-authored PRs originate from contributors without prior code ownership.<n>In contrast to human-created PRs where non-owner developers receive the most feedback, AI-co-authored PRs from non-owners receive the least.
  arXiv  Detail & Related papers  (2026-01-20T09:09:53Z)
• LazyReview A Dataset for Uncovering Lazy Thinking in NLP Peer Reviews [74.87393214734114]
  This work introduces LazyReview, a dataset of peer-review sentences annotated with fine-grained lazy thinking categories.<n>Large Language Models (LLMs) struggle to detect these instances in a zero-shot setting.<n> instruction-based fine-tuning on our dataset significantly boosts performance by 10-20 performance points.
  arXiv  Detail & Related papers  (2025-04-15T10:07:33Z)
• Can LLM feedback enhance review quality? A randomized study of 20K reviews at ICLR 2025 [115.86204862475864]
  Review Feedback Agent provides automated feedback on vague comments, content misunderstandings, and unprofessional remarks to reviewers.<n>It was implemented at ICLR 2025 as a large randomized control study.<n> 27% of reviewers who received feedback updated their reviews, and over 12,000 feedback suggestions from the agent were incorporated by those reviewers.
  arXiv  Detail & Related papers  (2025-04-13T22:01:25Z)
• Understanding the Helpfulness of Stale Bot for Pull-based Development: An Empirical Study of 20 Large Open-Source Projects [5.0760915931572]
  Stale bot was introduced by GitHub to automatically track, follow up, and close such inactive PRs. We perform an empirical study of 20 large and popular open-source projects. We find that Stale bot can help deal with a backlog of unresolved PRs as the projects closed more PRs within the first few months of adoption. However, Stale bot can also negatively affect the contributors as the projects experienced a considerable decrease in their number of active contributors after the adoption.
  arXiv  Detail & Related papers  (2023-05-29T15:25:06Z)
• Suggestion Bot: Analyzing the Impact of Automated Suggested Changes on Code Reviews [2.773900417167691]
  We created a bot called SUGGESTION BOT to automatically review the code base using GitHub's suggested changes functionality. We evaluate SUGGESTION BOT concerning its impact on review time and also analyze whether the comments given by the bot are clear and useful for users.
  arXiv  Detail & Related papers  (2023-05-10T17:33:43Z)
• A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments [70.1864008701113]
  Bots are used in Github repositories to automate repetitive activities that are part of the distributed software development process. This paper proposes a ground-truth dataset, based on a manual analysis with high interrater agreement, of pull request and issue comments in 5,000 distinct Github accounts. We propose an automated classification model to detect bots, taking as main features the number of empty and non-empty comments of each account, the number of comment patterns, and the inequality between comments within comment patterns.
  arXiv  Detail & Related papers  (2020-10-07T09:30:52Z)
• Automating App Review Response Generation [67.58267006314415]
  We propose a novel approach RRGen that automatically generates review responses by learning knowledge relations between reviews and their responses. Experiments on 58 apps and 309,246 review-response pairs highlight that RRGen outperforms the baselines by at least 67.4% in terms of BLEU-4.
  arXiv  Detail & Related papers  (2020-02-10T05:23:38Z)
• Perception and Acceptance of an Autonomous Refactoring Bot [11.908989544044998]
  We deployed an autonomous bot for 41 days in a student software development project. We conducted semi-structured interviews to find out how developers perceive the bot and whether they are more or less critical when reviewing the contributions of a bot compared to human contributions. Our findings show that the bot was perceived as a useful and unobtrusive contributor, and developers were no more critical of it than they were about their human colleagues, but only a few team members felt responsible for the bot.
  arXiv  Detail & Related papers  (2020-01-08T14:47:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.