Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code

• URL: http://arxiv.org/abs/2606.22721v1
• Date: Sun, 21 Jun 2026 23:38:33 GMT
• Title: Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code
• Abstract summary: We study repeat reviewers who collectively reviewed over a seven-month observation period.<n>We observe a population-level shift in approval rate from 30.1% to 36.8%.<n>This shift is experience-driven (persists after controlling for calendar time), agent-specific latency (human PR approval rates decline over the same period), and not explained by PR difficulty (median PR size is flat).<n>The combination of rising approval, declining comment effort, and increasing queue time is most consistent with reflexive habituation under growing workload rather than rational trust calibration alone.
• Score: 7.362328661065831
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As AI coding agents (e.g., GitHub Copilot, Devin, OpenAI Codex, Cursor) submit pull requests to open-source repositories at scale, a key question arises: do human reviewers gradually lower their scrutiny for AI-generated code over time? We conduct a longitudinal within-reviewer analysis using the AIDev dataset, studying 400 repeat reviewers who collectively submitted 11,429 reviews over a seven-month observation period. Comparing each reviewer's early and late review episodes, we observe a population-level shift in approval rate from 30.1% to 36.8% (Wilcoxon signed-rank p < 10^{-6} on paired shifts). Pooled by within-reviewer experience decile, the cumulative gap reaches +14.5 pp from first to tenth decile. This shift is experience-driven (persists after controlling for calendar time), agent-specific (human PR approval rates decline over the same period), and not explained by PR difficulty (median PR size is flat). However, review latency increases rather than decreases (+3.5x), while inline comment volume decreases (-22%, p=0.0014), suggesting reviewers spend more time in queue but less time actively inspecting code. The combination of rising approval, declining comment effort, and increasing queue time is most consistent with reflexive habituation under growing workload rather than rational trust calibration alone.

Related papers

• Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review [2.9152186617858304]
  We study 11,429 reviews paired with 10,104 human-authored inline comments from AI.<n>Approval rates rise from 30.5% in reviewers' early periods to 36.6% in late periods.<n>Four hand-crafted linguistic features - lexical diversity, Shannon entropy, technical specificity, and constructive actionability - show no monotonic decline across exposure deciles.
  arXiv  Detail & Related papers  (2026-09-05T18:15:18Z)
• AI-to-AI Code Reviews of GitHub Pull Requests [0.0]
  Cross-product AI-to-AI review occurred in approximately 1.6% of identified agent-authored PRs.<n>For three of four dual-role reviewers, mean comments per PR were 58-65% higher in the same-product group.<n>Overall, closed-loop AI-to-AI review is increasing but remains a minority of identified agent activity.
  arXiv  Detail & Related papers  (2026-08-21T17:17:35Z)
• SWE-Review: Closing the Loop on Issue Resolution with Agentic Code Review [47.72392142544407]
  Coding agents increasingly generate pull requests (PRs) for real-world software issues.<n>One-shot PR generation remains open-loop: the PR is proposed without systematic review, diagnosis, or revision.<n>We introduce textbfSWE-Review, a framework for closing this loop with agentic code review.
  arXiv  Detail & Related papers  (2026-07-07T09:37:45Z)
• Rethinking Code Review in the Age of AI: A Vision for Agentic Code Review [2.2807488730605296]
  We present a vision for an AI-powered code review workflow combining specialized agents with human-controlled quality.<n>We identify major open challenges for responsible adoption, including reliability, bias, privacy, automation bias, transparency, and evaluation.
  arXiv  Detail & Related papers  (2026-05-17T17:04:21Z)
• Stop Automating Peer Review Without Rigorous Evaluation [51.53099943385505]
  This position paper argues that today's AI systems should not be used to produce paper reviews.<n>We identify two critical issues: 1) AI reviewers exhibit a hivemind effect of excessive agreement within and across papers that reduces perspective diversity.<n>We argue that addressing the peer review crisis requires a science of peer review automation.
  arXiv  Detail & Related papers  (2026-05-04T22:41:04Z)
• These Aren't the Reviews You're Looking For How Humans Review AI-Generated Pull Requests [0.0]
  We analyze code review interactions for AI-generated pull requests (PRs) on GitHub using the AIDev dataset.<n>We find that most AI-generated PRs receive no review and, when reviewed, are largely dominated by AI agents rather than humans.
  arXiv  Detail & Related papers  (2026-05-04T06:32:50Z)
• On the Footprints of Reviewer Bots Feedback on Agentic Pull Requests in OSS GitHub Repositories [2.155083987188202]
  This study investigates the relationship between reviewer-bot feedback and PR outcomes.<n>We analyze 7,416 reviewer-bot comments on 4,532 PRs from the AI_Dev dataset.<n>Our results show that reviewer-bot comments mainly focus on bug fixes, testing, and documentation, are civil in tone, and are prescriptive in nature.
  arXiv  Detail & Related papers  (2026-04-27T13:17:13Z)
• When AI reviews science: Can we trust the referee? [73.47745294608072]
  We map attacks across the review lifecycle -- training and data retrieval, desk review, deep review, rebuttal, and system-level.<n>We use two advanced LLM-based referees to isolate the causal effects of prestige framing, assertion strength, rebuttal sycophancy, and contextual poisoning on review scores.
  arXiv  Detail & Related papers  (2026-04-26T08:03:32Z)
• SWE-chat: Coding Agent Interactions From Real Users in the Wild [70.18158706281724]
  SWE-chat is the first large-scale dataset of real coding agent sessions collected from open-source developers in the wild.<n>The dataset currently contains 6,000 sessions, comprising more than 63,000 user prompts and 355,000 agent tool calls.
  arXiv  Detail & Related papers  (2026-04-22T17:08:19Z)
• Early-Stage Prediction of Review Effort in AI-Generated Pull Requests [0.0]
  We analyze 33,707 agent-authored PRs from the AIDev dataset across 2,807 repositories.<n>We propose a Circuit Breaker triage model that predicts high-review-effort PRs at creation time.
  arXiv  Detail & Related papers  (2026-01-02T17:18:01Z)
• Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation [87.47155146067962]
  We provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of tasks.<n>We conduct three-dimensional analysis spanning models, scaffolds, and benchmarks.<n>Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs.
  arXiv  Detail & Related papers  (2025-10-13T22:22:28Z)
• CoCoNUTS: Concentrating on Content while Neglecting Uninformative Textual Styles for AI-Generated Peer Review Detection [60.52240468810558]
  We introduce CoCoNUTS, a content-oriented benchmark built upon a fine-grained dataset of AI-generated peer reviews.<n>We also develop CoCoDet, an AI review detector via a multi-task learning framework, to achieve more accurate and robust detection of AI involvement in review content.
  arXiv  Detail & Related papers  (2025-08-28T06:03:11Z)
• Automated Code Review In Practice [1.6271516689052665]
  Several AI-assisted tools, such as Qodo, GitHub Copilot, and Coderabbit, provide automated reviews using large language models (LLMs)<n>This study examines the impact of LLM-based automated code review tools in an industrial setting.
  arXiv  Detail & Related papers  (2024-12-24T16:24:45Z)
• Automating App Review Response Generation [67.58267006314415]
  We propose a novel approach RRGen that automatically generates review responses by learning knowledge relations between reviews and their responses. Experiments on 58 apps and 309,246 review-response pairs highlight that RRGen outperforms the baselines by at least 67.4% in terms of BLEU-4.
  arXiv  Detail & Related papers  (2020-02-10T05:23:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.