These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests

• URL: http://arxiv.org/abs/2605.02273v1
• Date: Mon, 04 May 2026 06:32:50 GMT
• Title: These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests
• Abstract summary: We analyze code review interactions for AI-generated pull requests (PRs) on GitHub using the AIDev dataset.<n>We find that most AI-generated PRs receive no review and, when reviewed, are largely dominated by AI agents rather than humans.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We analyze code review interactions for AI-generated pull requests (PRs) on GitHub using the AIDev dataset and compare them to human-authored PRs within the same repositories. We find that most AI-generated PRs receive no review and, when reviewed, are largely dominated by AI agents rather than humans. Human-authored PRs are more likely to receive human-only review and to attract direct human feedback. In contrast, reviews of AI-generated PRs more often take the form of automation-mediated interaction, with human involvement frequently expressed through agent steering rather than standalone evaluation. These results indicate systematic differences in how review activity is structured in agentic workflows and raise challenges for interpreting review metrics as indicators of human oversight in large-scale mining studies.

Related papers

• From Human-Centric to Agentic Code Review: The Impact of Different Generations of Generative AI Technology on Review Quality [3.7086626614863984]
  We study 1.02 million reviewed pull requests from 207 GitHub projects that transition across three code review eras.<n>Agent-involved collaboration patterns are associated with faster review decisions under Gradual AI Adoption and Rapid AI Agent Adoption.<n>These findings provide empirical guidance for designing AI-supported code review processes that improve efficiency without weakening review quality.
  arXiv  Detail & Related papers  (2026-07-14T18:48:20Z)
• Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review [2.2807488730605296]
  We present a vision for an AI-powered code review workflow combining specialized agents with human-controlled quality.<n>We identify major open challenges for responsible adoption, including reliability, bias, privacy, automation bias, transparency, and evaluation.
  arXiv  Detail & Related papers  (2026-05-17T17:04:21Z)
• Stop Automating Peer Review Without Rigorous Evaluation [51.53099943385505]
  This position paper argues that today's AI systems should not be used to produce paper reviews.<n>We identify two critical issues: 1) AI reviewers exhibit a hivemind effect of excessive agreement within and across papers that reduces perspective diversity.<n>We argue that addressing the peer review crisis requires a science of peer review automation.
  arXiv  Detail & Related papers  (2026-05-04T22:41:04Z)
• On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories [2.155083987188202]
  This study investigates the relationship between reviewer-bot feedback and PR outcomes.<n>We analyze 7,416 reviewer-bot comments on 4,532 PRs from the AI_Dev dataset.<n>Our results show that reviewer-bot comments mainly focus on bug fixes, testing, and documentation, are civil in tone, and are prescriptive in nature.
  arXiv  Detail & Related papers  (2026-04-27T13:17:13Z)
• AI-Assisted Peer Review at Scale: The AAAI-26 AI Review Pilot [45.90272517740283]
  We report the first large-scale field deployment of AI-assisted peer review.<n>Every main-track submission at AAAI-26 received one clearly identified AI review from a state-of-the-art system.
  arXiv  Detail & Related papers  (2026-04-15T14:51:07Z)
• Humans Integrate, Agents Fix: How Agent-Authored Pull Requests Are Referenced in Practice [4.359007050137919]
  We introduce a taxonomy to characterize the intent of agent-authored PR references across Human-to-Agent and Agent-to-Agent interactions.<n>Our analysis shows that while humans initiate most references to agent-authored PRs, a substantial portion of these interactions are AI-assisted.<n>We find that referencing/referenced PRs are associated with substantially longer lifespan and review times compared to isolated PRs.
  arXiv  Detail & Related papers  (2026-04-05T11:04:46Z)
• How AI Coding Agents Communicate: A Study of Pull Request Description Characteristics and Human Review Responses [6.061536429904841]
  We conduct an empirical analysis of pull requests created by five AI coding agents using the AIDev dataset.<n>We find that AI coding agents exhibit distinct PR description styles, which are associated with differences in reviewer engagement, response time, and merge outcomes.
  arXiv  Detail & Related papers  (2026-02-19T05:06:31Z)
• On Autopilot? An Empirical Study of Human-AI Teaming and Review Practices in Open Source [11.412808537439973]
  We investigated project-level guidelines and developers' interactions with AI-assisted pull requests (PRs)<n>We found that over 67.5% of AI-co-authored PRs originate from contributors without prior code ownership.<n>In contrast to human-created PRs where non-owner developers receive the most feedback, AI-co-authored PRs from non-owners receive the least.
  arXiv  Detail & Related papers  (2026-01-20T09:09:53Z)
• Early-Stage Prediction of Review Effort in AI-Generated Pull Requests [0.0]
  We analyze 33,707 agent-authored PRs from the AIDev dataset across 2,807 repositories.<n>We propose a Circuit Breaker triage model that predicts high-review-effort PRs at creation time.
  arXiv  Detail & Related papers  (2026-01-02T17:18:01Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• Aligning Large Language Models from Self-Reference AI Feedback with one General Principle [61.105703857868775]
  We propose a self-reference-based AI feedback framework that enables a 13B Llama2-Chat to provide high-quality feedback. Specifically, we allow the AI to first respond to the user's instructions, then generate criticism of other answers based on its own response as a reference. Finally, we determine which answer better fits human preferences according to the criticism.
  arXiv  Detail & Related papers  (2024-06-17T03:51:46Z)
• Beyond Static Evaluation: A Dynamic Approach to Assessing AI Assistants' API Invocation Capabilities [48.922660354417204]
  We propose Automated Dynamic Evaluation (AutoDE) to assess an assistant's API call capability without human involvement. In our framework, we endeavor to closely mirror genuine human conversation patterns in human-machine interactions.
  arXiv  Detail & Related papers  (2024-03-17T07:34:12Z)
• Exploration with Principles for Diverse AI Supervision [88.61687950039662]
  Training large transformers using next-token prediction has given rise to groundbreaking advancements in AI. While this generative AI approach has produced impressive results, it heavily leans on human supervision. This strong reliance on human oversight poses a significant hurdle to the advancement of AI innovation. We propose a novel paradigm termed Exploratory AI (EAI) aimed at autonomously generating high-quality training data.
  arXiv  Detail & Related papers  (2023-10-13T07:03:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.