LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails

• URL: http://arxiv.org/abs/2609.02246v1
• Date: Wed, 02 Sep 2026 07:54:23 GMT
• Title: LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails
• Abstract summary: Self-improving agent pipelines have a problem at their center.<n>An evaluation signal failed in eleven classes: judge bias, harness and metric failures, ground-truth errors, and reward hacking.<n>We describe PROCTOR, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-improving agent pipelines have a problem at their center. An optimizer rewrites prompts to score higher, and the score comes from a judge that is itself an LLM. That judge has the last word on whether the system is getting better, and our position is that it has not earned it. The judge should be demoted from oracle to advisor: its verdict becomes one input among several, and every change is gated instead by a deterministic verification layer the judge cannot override. We reached this position by building the alternative and running it. Over months of running autonomous prompt-optimization loops in production across contract analysis, compliance review, and code quality, we cataloged eleven ways the evaluation signal failed, in four classes: judge bias, harness and metric failures, ground-truth errors, and reward hacking. Agents achieved perfect scores by reading cached answer keys from their environment, a 100% pass rate concealing 68% true capability. A corrupted ground-truth label caused the optimizer to delete correct compliance rules to agree with it. A syntactically broken prompt was promoted as the winner because a silent parser fallback improved the metric. Attempts to fix the judge by rewriting its rubric plateaued; the only reliable gain came from a structural constraint on its output order. In response we describe PROCTOR, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply, and a Teacher grades those mutations under five deterministic guardrails: hermetic sandboxes, capability-disjoint roles, acceptance checks that outrank the Teacher, frozen holdouts, and canary cases engineered so that a perfect score is itself evidence of cheating. We report the failures this prevented, and, because the Teacher is itself an LLM judge, the failures it did not.

Related papers

• Conformity Breaks Conformal Prediction [1.3583317564926913]
  A conformal certificate can be valid when an LLM answers alone and invalid when the same LLM sees peers that unanimously assert a wrong answer.<n>We show that this shift silently breaks conformal prediction in multi-agent LLM systems.
  arXiv  Detail & Related papers  (2026-09-03T20:05:47Z)
• Commit-first LLM judging inherits the judge's own errors [0.0]
  LLM judges, models that score another system's output, can be gamed by the systems they score.<n>Recent work identifies one defence that works: the judge solves the task itself first and commits to that answer, then accepts a candidate only if the two match.<n>We call this commit-first judging, and ask whether shipped software implements it, and what it costs.
  arXiv  Detail & Related papers  (2026-08-31T12:14:14Z)
• trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories [0.0]
  Outcome-only evaluation is the production default for LLM agents.<n>We measure that blind spot where ground truth is known by construction.<n>We argue that judge evaluations must stratify recall by outcome survival.
  arXiv  Detail & Related papers  (2026-08-29T10:14:05Z)
• RecurSE: Bounded Recursive Self-Evaluation for LLM Rubric Judges [48.831446371402414]
  RecurSE pairs a trainable judge with a synchronized policy-copy checker that audits the judge's reasoning against meta-rubrics to supply a scalar process reward.<n>We show that RecurSE achieves consistent generalization gains across held-out medical, pairwise, summarization, and professional benchmarks.
  arXiv  Detail & Related papers  (2026-08-25T08:35:04Z)
• Competence, Not Accuracy: A Diagnostic for Reference-Free Judge Gates in Skill Optimization [9.9751102700286]
  Text-space skill optimization adapts a frozen agent by evolving a natural-language skill document, accepting each candidate through a validation gate.<n>We ask a prior question: can we tell, before placing a judge in the loop, whether its scores separate correct from incorrect answers at all?<n>We formalize a reference-free judge as a latent solver -- its verdict rests on agreement with whatever it would itself conclude, so its capacity to evaluate is bounded by its capacity to solve.<n>We find discriminability at chance where competence sits near the floor and usable above it; that a judge's benchmark accuracy overstates the competence that
  arXiv  Detail & Related papers  (2026-08-19T09:19:23Z)
• Teaching a Large Language Model Tutor to Withhold the Answer: A Supervisor Architecture and an Evidence-Driven Method for Tuning Socratic Behavior [0.0]
  An effective large language model (LLM) tutor must often decline to give an answer it could easily produce.<n>We report a deployed tutoring system that enforces answer-withholding as a per-turn, machine-checkable contract.<n>We tune the behavior with an automated evaluation that uses no human subjects.
  arXiv  Detail & Related papers  (2026-08-12T17:35:58Z)
• LivingArena: Do LLMs Know What Other LLMs Don't? Peer-Probing as Scalable Evaluation [79.03892269184145]
  LivingArena is an automated, contamination-resistant evaluation framework.<n>Models take turns proposing questions, aiming to pose items that opponents cannot answer correctly.<n>Questioners are encouraged to actively identify and exploit opponents' knowledge boundaries.
  arXiv  Detail & Related papers  (2026-06-19T06:20:58Z)
• Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges [65.92623263645139]
  We introduce BITE, a black-box adversarial framework that learns semantics-preserving edits to mislead an LLM judge.<n>BITE achieves an attack success rate exceeding 65% and raises scores by 1-2 points on a 9-point scale.<n>Our findings expose a fundamental weakness in the LLM-as-a-judge paradigm and motivate robust, attack-aware evaluation.
  arXiv  Detail & Related papers  (2026-05-24T05:24:09Z)
• Bias in the Loop: Auditing LLM-as-a-Judge for Software Engineering [0.0]
  Large Language Models are increasingly used as judges to evaluate code artifacts.<n>Current practice lacks a principled account of reliability and bias.<n>This paper studies LLM-as-a-Judge for code through a measurement-first lens.
  arXiv  Detail & Related papers  (2026-04-18T02:35:05Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• Conversation for Non-verifiable Learning: Self-Evolving LLMs through Meta-Evaluation [56.84819098277464]
  CoNL is a framework that unifies generation, evaluation, and meta-evaluation through multi-agent self-play.<n>CoNL achieves consistent improvements over self-rewarding baselines while maintaining stable training.
  arXiv  Detail & Related papers  (2026-01-29T09:41:14Z)
• Validating LLM-as-a-Judge Systems under Rating Indeterminacy [65.137380612741]
  We introduce a framework for validating LLM-as-a-judge systems under rating indeterminacy.<n>We demonstrate that differences in how humans and LLMs resolve rating indeterminacy when responding to forced-choice rating instructions can heavily bias validation.
  arXiv  Detail & Related papers  (2025-03-07T22:09:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.