Early-Stage Prediction of Review Effort in AI-Generated Pull Requests

• URL: http://arxiv.org/abs/2601.00753v1
• Date: Fri, 02 Jan 2026 17:18:01 GMT
• Title: Early-Stage Prediction of Review Effort in AI-Generated Pull Requests
• Authors: Dao Sy Duy Minh, Huynh Trung Kiet, Tran Chi Nguyen, Nguyen Lam Phu Quy, Phu Hoa Pham, Nguyen Dinh Ha Duong, Truong Bao Tran,
• Abstract summary: We analyze 33,707 agent-authored PRs from the AIDev dataset across 2,807 repositories.<n>We propose a Circuit Breaker triage model that predicts high-review-effort PRs at creation time.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As autonomous AI agents transition from code completion tools to full-fledged teammates capable of opening pull requests (PRs) at scale, software maintainers face a new challenge: not just reviewing code, but managing complex interaction loops with non-human contributors. This paradigm shift raises a critical question: can we predict which agent-generated PRs will consume excessive review effort before any human interaction begins? Analyzing 33,707 agent-authored PRs from the AIDev dataset across 2,807 repositories, we uncover a striking two-regime behavioral pattern that fundamentally distinguishes autonomous agents from human developers. The first regime, representing 28.3 percent of all PRs, consists of instant merges (less than 1 minute), reflecting success on narrow automation tasks. The second regime involves iterative review cycles where agents frequently stall or abandon refinement (ghosting). We propose a Circuit Breaker triage model that predicts high-review-effort PRs (top 20 percent) at creation time using only static structural features. A LightGBM model achieves AUC 0.957 on a temporal split, while semantic text features (TF-IDF, CodeBERT) provide negligible predictive value. At a 20 percent review budget, the model intercepts 69 percent of total review effort, enabling zero-latency governance. Our findings challenge prevailing assumptions in AI-assisted code review: review burden is dictated by what agents touch, not what they say, highlighting the need for structural governance mechanisms in human-AI collaboration.

Related papers

• AgentIR: Reasoning-Aware Retrieval for Deep Research Agents [76.29382561831105]
  Deep Research agents generate explicit natural language reasoning before each search call.<n> Reasoning-Aware Retrieval embeds the agent's reasoning trace alongside its query.<n>DR- Synth generates Deep Research retriever training data from standard QA datasets.<n>AgentIR-4B achieves 68% accuracy with the open-weight agent Tongyi-DeepResearch.
  arXiv  Detail & Related papers  (2026-03-04T18:47:26Z)
• How AI Coding Agents Communicate: A Study of Pull Request Description Characteristics and Human Review Responses [6.061536429904841]
  We conduct an empirical analysis of pull requests created by five AI coding agents using the AIDev dataset.<n>We find that AI coding agents exhibit distinct PR description styles, which are associated with differences in reviewer engagement, response time, and merge outcomes.
  arXiv  Detail & Related papers  (2026-02-19T05:06:31Z)
• Why Are AI Agent Involved Pull Requests (Fix-Related) Remain Unmerged? An Empirical Study [5.127121704630949]
  We analyze 8,106 fix related PRs authored by five widely used AI coding agents from the AIDEV POP dataset.<n>Our results indicate that test case failures and prior resolution of the same issues by other PRs are the most common causes of non integration.
  arXiv  Detail & Related papers  (2026-01-29T22:06:58Z)
• AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios [49.90735676070039]
  The capacity of AI agents to effectively handle tasks of increasing duration and complexity continues to grow.<n>We argue that current evaluations prioritize increasing task difficulty without sufficiently addressing the diversity of agentic tasks.<n>We propose AgentIF-OneDay, aimed at determining whether general users can utilize natural language instructions and AI agents to complete a diverse array of daily tasks.
  arXiv  Detail & Related papers  (2026-01-28T13:49:18Z)
• Let's Make Every Pull Request Meaningful: An Empirical Analysis of Developer and Agentic Pull Requests [0.944838645453772]
  We conduct a large-scale empirical analysis of 40,214 PRs collected from the AIDev dataset.<n>We extract 64 features across six families and fit statistical regression models to compare PR merge outcomes for human and agentic PRs.<n>Our results show that submitter attributes dominate merge outcomes for both groups, while review-related features exhibit contrasting effects between human and agentic PRs.
  arXiv  Detail & Related papers  (2026-01-26T18:16:10Z)
• On Autopilot? An Empirical Study of Human-AI Teaming and Review Practices in Open Source [11.412808537439973]
  We investigated project-level guidelines and developers' interactions with AI-assisted pull requests (PRs)<n>We found that over 67.5% of AI-co-authored PRs originate from contributors without prior code ownership.<n>In contrast to human-created PRs where non-owner developers receive the most feedback, AI-co-authored PRs from non-owners receive the least.
  arXiv  Detail & Related papers  (2026-01-20T09:09:53Z)
• AI IDEs or Autonomous Agents? Measuring the Impact of Coding Agents on Software Development [12.50615284537175]
  Large language model (LLM) based coding agents increasingly act as autonomous contributors that generate and merge pull requests.<n>We present a longitudinal causal study of agent adoption in open-source repositories using staggered difference-in-differences with matched controls.
  arXiv  Detail & Related papers  (2026-01-20T04:51:56Z)
• Security in the Age of AI Teammates: An Empirical Study of Agentic Pull Requests on GitHub [4.409447722044799]
  This study aims to characterize how autonomous coding agents contribute to software security in practice.<n>We conduct a large-scale empirical analysis of agent-authored PRs using the AIDev dataset.<n>We then analyze prevalence, acceptance outcomes, and review latency across autonomous agents, programming ecosystems, and types of code changes.
  arXiv  Detail & Related papers  (2026-01-01T21:14:11Z)
• Towards a Science of Scaling Agent Systems [79.64446272302287]
  We formalize a definition for agent evaluation and characterize scaling laws as the interplay between agent quantity, coordination structure, modelic, and task properties.<n>We derive a predictive model using coordination metrics, that cross-validated R2=0, enabling prediction on unseen task domains.<n>We identify three effects: (1) a tool-coordination trade-off: under fixed computational budgets, tool-heavy tasks suffer disproportionately from multi-agent overhead, and (2) a capability saturation: coordination yields diminishing or negative returns once single-agent baselines exceed 45%.
  arXiv  Detail & Related papers  (2025-12-09T06:52:21Z)
• Towards a Humanized Social-Media Ecosystem: AI-Augmented HCI Design Patterns for Safety, Agency & Well-Being [0.0]
  Social platforms connect billions of people, yet their engagement-first algorithms often work on users rather than with them.<n>We propose Human-Layer AI (HL-AI)--user-owned, explainable intermediaries that sit in the browser between platform logic and the interface.<n>HL-AI gives people practical, moment-to-moment control without requiring platform cooperation.
  arXiv  Detail & Related papers  (2025-11-08T06:22:15Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• Impatient Users Confuse AI Agents: High-fidelity Simulations of Human Traits for Testing Agents [58.00130492861884]
  TraitBasis is a lightweight, model-agnostic method for systematically stress testing AI agents.<n>TraitBasis learns directions in activation space corresponding to steerable user traits.<n>We observe on average a 2%-30% performance degradation on $tau$-Trait across frontier models.
  arXiv  Detail & Related papers  (2025-10-06T05:03:57Z)
• On the Use of Agentic Coding: An Empirical Study of Pull Requests on GitHub [6.7302091035327285]
  Large language models (LLMs) are increasingly being integrated into software development processes.<n>The ability to generate code and submit pull requests with minimal human intervention, through the use of autonomous AI agents, is poised to become a standard practice.<n>We empirically study 567 GitHub pull requests (PRs) generated using Claude Code, an agentic coding tool, across 157 open-source projects.
  arXiv  Detail & Related papers  (2025-09-18T08:48:32Z)
• SFR-DeepResearch: Towards Effective Reinforcement Learning for Autonomously Reasoning Single Agents [93.26456498576181]
  This paper focuses on the development of native Autonomous Single-Agent models for Deep Research.<n>Our best variant SFR-DR-20B achieves up to 28.7% on Humanity's Last Exam benchmark.
  arXiv  Detail & Related papers  (2025-09-08T02:07:09Z)
• Code with Me or for Me? How Increasing AI Automation Transforms Developer Workflows [60.04362496037186]
  We present the first controlled study of developer interactions with coding agents.<n>We evaluate two leading copilot and agentic coding assistants.<n>Our results show agents can assist developers in ways that surpass copilots.
  arXiv  Detail & Related papers  (2025-07-10T20:12:54Z)
• Generative AI for Pull Request Descriptions: Adoption, Impact, and Developer Interventions [11.620351603683496]
  GitHub's Copilot for Pull Requests (PRs) is a promising service aiming to automate various developer tasks related to PRs. In this study, we examine 18,256 PRs in which parts of the descriptions were crafted by generative AI. Our findings indicate that Copilot for PRs, though in its infancy, is seeing a marked uptick in adoption.
  arXiv  Detail & Related papers  (2024-02-14T06:20:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.