Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines

• URL: http://arxiv.org/abs/2606.15474v1
• Date: Sat, 13 Jun 2026 21:12:27 GMT
• Title: Who Drifted: the System or the Judge? Anytime-Valid Attribution in LLM Evaluation Pipelines
• Abstract summary: Continuous evaluation of products relies on a strong LLM judge treated as ground truth.<n>Judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores.<n>We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard.
• Score: 0.22843885788439797
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continuous evaluation of LLM products relies on a strong LLM judge treated as ground truth: a cheap monitor scores every interaction and a team is paged when the score drifts down. But the judge is itself a model behind an API, and a silent version bump or scoring-prompt update changes how it scores -- so every drift alarm is ambiguous between a worse product and a changed judge. We resolve the ambiguity with a fixed, human-labeled anchor set that the current judge re-scores at a steady interleave, a second betting e-process on the judge-versus-human gap, and a guard-window rule returning a verdict in {none, system, judge}. We prove anytime-validity, one-way identification (only the judge can move the anchors), an attribution race whose design law is that the anchors must out-run the main process they guard, and process orthogonality. On two real judge changes, a silent version bump is detected as judge drift in 60/60 runs with zero judge-to-system misattribution, and a contaminating strict-prompt change is correctly attributed on 110 of 120 runs at guard width 300 -- while the industry-default rolling z-test false-alarms on 75% of drift-free streams. Every experiment replicates on a second domain (TL;DR summarization) with nothing re-tuned, and where the domains differ the differences are the ones the race predicts: the strict-prompt change shifts scores harder there, so the anchors fire faster and attribution becomes perfect (240/240). The monitor runs at approximately 0.64 of the cost of strong-judging every item, or 0.21 in a cheaper-but-deafer regime.

Related papers

• LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails [0.0]
  Self-improving agent pipelines have a problem at their center.<n>An evaluation signal failed in eleven classes: judge bias, harness and metric failures, ground-truth errors, and reward hacking.<n>We describe PROCTOR, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply.
  arXiv  Detail & Related papers  (2026-09-02T07:54:23Z)
• trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories [0.0]
  Outcome-only evaluation is the production default for LLM agents.<n>We measure that blind spot where ground truth is known by construction.<n>We argue that judge evaluations must stratify recall by outcome survival.
  arXiv  Detail & Related papers  (2026-08-29T10:14:05Z)
• Difference-in-Differences on a Censored Rating Scale Can Manufacture an Effect: Evidence from a Pre-Registered LLM-Judge Audit [15.81584544074437]
  Audits of LLM judges certify a bias by contrasting matched conditions.<n>We show that this endpoint is not identified on the scale that reports it.<n>We derive the mechanism in closed form and show that its contribution is measurable from an audit's own ratings.
  arXiv  Detail & Related papers  (2026-08-27T16:10:16Z)
• RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges [48.831446371402414]
  RecurSE pairs a trainable judge with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward.<n>We show that RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks.
  arXiv  Detail & Related papers  (2026-08-25T08:35:04Z)
• More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judges [4.167333498582309]
  Training a language model against its own reference-free judgments assumes a model's verdict on a shown answer tracks correctness.<n>We show it fails structurally: conditioned on a candidate, a judge scores plausibility, not correctness, leaving false-positive basins a policy learns to exploit.
  arXiv  Detail & Related papers  (2026-07-07T06:59:30Z)
• Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges [65.92623263645139]
  We introduce BITE, a black-box adversarial framework that learns semantics-preserving edits to mislead an LLM judge.<n>BITE achieves an attack success rate exceeding 65% and raises scores by 1-2 points on a 9-point scale.<n>Our findings expose a fundamental weakness in the LLM-as-a-judge paradigm and motivate robust, attack-aware evaluation.
  arXiv  Detail & Related papers  (2026-05-24T05:24:09Z)
• Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges [3.974554103907186]
  We show that in multi-judge evaluation with labeled calibration data, do not discard weak judges by accuracy alone.<n>We explain this reversal with oracle analyses showing that the optimal calibrated risk under proper scoring rules cannot increase when additional judge signals are made available.
  arXiv  Detail & Related papers  (2026-05-10T18:49:58Z)
• Two Ways to De-Bias an LLM-as-a-Judge: A Continuous-Score Comparison of Hierarchical Bayesian Calibration and Neural-ODE Score Transport [0.0]
  We compare two correctors that take opposing views on how this mapping should be modeled.<n>We translate these findings into an explicit decision rule for production deployments.
  arXiv  Detail & Related papers  (2026-05-09T23:55:02Z)
• How Sensitive Are Safety Benchmarks to Judge Configuration Choices? [5.94231111588812]
  Safety benchmarks such as HarmBench rely on judges to classify model responses as harmful or safe.<n>We show that the combination of judge model and judge prompt is typically treated as a fixed implementation detail.<n>We find that prompt wording alone, holding the judge model fixed, shifts measured harmful-response rates by up to 24.2 percentage points.
  arXiv  Detail & Related papers  (2026-04-27T05:59:59Z)
• C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning [0.6138671548064355]
  Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning.<n>We introduce C2-Faith, a benchmark that targets two complementary dimensions of faithfulness: causality (does each step logically follow from prior context?) and coverage.<n>We evaluate three frontier judges under three tasks: binary causal detection, causal step localization, and coverage scoring.
  arXiv  Detail & Related papers  (2026-03-05T13:36:47Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• Evaluative Fingerprints: Stable and Systematic Differences in LLM Evaluator Behavior [0.0]
  We find that judges are consistent, but not with each other; they are consistent with themselves.<n>Across 3,240 evaluations, inter-judge agreement is near-zero.<n>Averaging judges' scores produces a synthetic verdict that corresponds to no judge's actual values.
  arXiv  Detail & Related papers  (2026-01-08T17:02:22Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Validating LLM-as-a-Judge Systems under Rating Indeterminacy [65.137380612741]
  We introduce a framework for validating LLM-as-a-judge systems under rating indeterminacy.<n>We demonstrate that differences in how humans and LLMs resolve rating indeterminacy when responding to forced-choice rating instructions can heavily bias validation.
  arXiv  Detail & Related papers  (2025-03-07T22:09:47Z)
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges [48.053949045598344]
  evaluating Large Language Models (LLMs) in open-ended scenarios is challenging because existing benchmarks and metrics can not measure them comprehensively.<n>We propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks.<n>We train JudgeLM at different scales from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its capabilities and behaviors.
  arXiv  Detail & Related papers  (2023-10-26T17:48:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.