Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review

• URL: http://arxiv.org/abs/2609.06213v1
• Date: Sat, 05 Sep 2026 18:15:18 GMT
• Title: Beyond Lexical Metrics: Sentence-Embedding Detection of Reviewer Habituation in AI Code Review
• Abstract summary: We study 11,429 reviews paired with 10,104 human-authored inline comments from AI.<n>Approval rates rise from 30.5% in reviewers' early periods to 36.6% in late periods.<n>Four hand-crafted linguistic features - lexical diversity, Shannon entropy, technical specificity, and constructive actionability - show no monotonic decline across exposure deciles.
• Score: 2.9152186617858304
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Code review is a key quality checkpoint between AI-generated code and production. As AI coding agents submit pull requests at scale, it is unclear whether reviewers reduce scrutiny with repeated exposure and whether review comments reveal this change. We study 11,429 reviews from 400 repeat reviewers over 207 days, paired with 10,104 human-authored inline comments from AIDev. Approval rates rise from 30.5% in reviewers' early periods to 36.6% in late periods (Wilcoxon p = 8.6 x 10^-8; Cohen's d = 0.25). However, four hand-crafted linguistic features - lexical diversity, Shannon entropy, technical specificity, and constructive actionability - show no monotonic decline across exposure deciles (all Spearman absolute rho <= 0.53, p >= 0.11; Bonferroni-corrected Mann-Whitney p >= 0.36). A logistic-regression classifier based on these features reaches F1 = 0.485, below the majority-class baseline. Sentence-embedding structure does carry signal: reviewers' late-period comment centroids shift farther from their early-period centroids than under within-reviewer random permutations (Wilcoxon p < 0.001), and a small MLP using three embedding statistics reaches F1 = 0.74 under 5-fold reviewer-stratified cross-validation. Granger analysis shows that approval-rate changes predict later shifts in technical specificity at all tested lags (p < 0.001), while the reverse direction is significant at only one of four lags. Reviewer adaptation is therefore detectable in latent distributional structure rather than in classical lexical metrics, and language shifts follow rather than precede changes in approval behavior.

Related papers

• From Code Review to Code Critique: Intent, Drift, and Spotlight for AI-Generated Diffs at Scale [2.8980676513012846]
  ARCTIC is an AI-powered Code Critique system that reframes code review around three capabilities: intent prediction, drift detection, and code spotlight.<n>In the experimental rollout, the drift scores reduces code misalignment by an additional 5.76 points (p = 0.026), intent prediction receives 90.2% approval, and zero defects have been attributed to self-reviewed diffs since launch.
  arXiv  Detail & Related papers  (2026-07-31T15:17:25Z)
• Habituation at the Gate: Rising Approval and Declining Scrutiny in Human Review of AI Agent Code [7.362328661065831]
  We study repeat reviewers who collectively reviewed over a seven-month observation period.<n>We observe a population-level shift in approval rate from 30.1% to 36.8%.<n>This shift is experience-driven (persists after controlling for calendar time), agent-specific latency (human PR approval rates decline over the same period), and not explained by PR difficulty (median PR size is flat).<n>The combination of rising approval, declining comment effort, and increasing queue time is most consistent with reflexive habituation under growing workload rather than rational trust calibration alone.
  arXiv  Detail & Related papers  (2026-06-21T23:38:33Z)
• Anticipating the Optimism Gap: Predicting Distribution-Shift Degradation of RF-Impairment Detectors from In-Distribution Statistics [0.0]
  Detectors for radio-frequency impairments are usually reported with a single AUC measured on the distribution they were tuned on.<n>We ask whether this optimism can be predicted before any out-of-distribution data is seen.
  arXiv  Detail & Related papers  (2026-06-20T14:10:59Z)
• FirstPass: Grounding AI Scientific Judgment in Multi-Round Editorial Outcomes [0.27998963147546146]
  We introduce FirstPass, a dataset fine-tuned model that addresses all three fronts of AI for peer review.<n>We exploit mandatory transparent peer review (instituted November 2022) and verify 100% content integrity by automated audit.<n>On generation, FirstPass produces reviews averaging 1,187 words, substantially closer to human references (2,155 words) than any baseline.
  arXiv  Detail & Related papers  (2026-06-18T15:06:36Z)
• Quickest Detection of Hallucination Onset: Delay Bounds and Learned CUSUM Statistics [0.0]
  hallucination onset detection is a quickest change detection problem.<n>A causal recurrent labeler acts as a CUSUM with a learned increment.<n>Low-false-alarm onset detection is hard.
  arXiv  Detail & Related papers  (2026-06-10T06:10:02Z)
• Catching One in Five: LLM-as-Judge Blind Spots in Production Multi-Turn Transaction Agents [45.148328075418156]
  We study a deployed multi-turn food-and-beverage ordering agent and measure how many genuine quality problems it catches.<n>Our blind-spot taxonomy shows the failure is structured, not random.<n>For production multi-turn agents, automated judging is a regression floor, not a substitute for human review.
  arXiv  Detail & Related papers  (2026-06-09T02:11:01Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Can Humans Tell? A Dual-Axis Study of Human Perception of LLM-Generated News [47.03825808787752]
  Can humans tell whether a news article was written by a person or a large language model (LLM)?<n>We investigate this question using JudgeGPT, a study platform that measures source attribution (human vs. machine) and authenticity judgment (legitimate vs. fake) on continuous scales.<n>We report five findings: (1) participants cannot reliably distinguish machine-generated from human-written text; (2) this inability holds across all tested models, including open-weight models with as few as 7B parameters; and (3) accuracy degrades after approximately 30 sequential evaluations due to cognitive fatigue.
  arXiv  Detail & Related papers  (2026-04-04T15:03:42Z)
• ORCA: Open-ended Response Correctness Assessment for Audio Question Answering [41.72231074041232]
  We present ORCA, a framework that models the variability in human judgments using Beta distributions to predict both expected correctness and uncertainty.<n>We collected 11,721 annotations across 3,580 question-answer pairs from 15 LALMs on two audio QA benchmarks, achieving inter-annotator agreement of 0.82 (Krippendorff's alpha)
  arXiv  Detail & Related papers  (2025-11-28T14:41:48Z)
• TARG: Training-Free Adaptive Retrieval Gating for Efficient RAG [46.122203287541005]
  Training-free Adaptive Retrieval Gating (TARG) is a single-shot policy that decides when to retrieve using only a short, no-latency draft from the base model.<n>On NQ-Open, TriviaQA, and PopQA, TARG consistently shifts the accuracy-efficiency frontier.
  arXiv  Detail & Related papers  (2025-11-12T23:09:52Z)
• Group-Adaptive Threshold Optimization for Robust AI-Generated Text Detection [58.419940585826744]
  We introduce FairOPT, an algorithm for group-specific threshold optimization for probabilistic AI-text detectors.<n>We partitioned data into subgroups based on attributes (e.g., text length and writing style) and implemented FairOPT to learn decision thresholds for each group to reduce discrepancy.<n>Our framework paves the way for more robust classification in AI-generated content detection via post-processing.
  arXiv  Detail & Related papers  (2025-02-06T21:58:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.