Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges

• URL: http://arxiv.org/abs/2605.06161v1
• Date: Thu, 07 May 2026 12:49:09 GMT
• Title: Beyond Accuracy: Policy Invariance as a Reliability Test for LLM Safety Judges
• Abstract summary: LLM-as-a-Judge pipelines have become the de facto evaluator for agent safety.<n>Existing benchmarks treat their verdicts as ground-truth proxies without checking whether the verdicts depend on the agent's behavior or merely on how the evaluation policy happens to be worded.<n>We operationalize it as three testable principles: rubric-semantics invariance under certified-equivalent rewrites, rubric-threshold invariance under intentional strict-to-lenient shifts, and ambiguity-aware calibration.
• Score: 26.595399077062638
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: LLM-as-a-Judge pipelines have become the de facto evaluator for agent safety, yet existing benchmarks treat their verdicts as ground-truth proxies without checking whether the verdicts depend on the agent's behavior or merely on how the evaluation policy happens to be worded. We argue that any trustworthy safety judge must satisfy a basic property we call policy invariance, and we operationalize it as three testable principles: rubric-semantics invariance under certified-equivalent rewrites, rubric-threshold invariance under intentional strict-to-lenient shifts, and ambiguity-aware calibration so that verdict instability concentrates on genuinely ambiguous cases. Instantiating these principles as a stress-test protocol with four agent-class judges on trajectories drawn from ASSEBench and R-Judge, we surface a previously unmeasured failure mode: today's judges respond to meaningful normative shifts and to meaningless structural rewrites with comparable strength, and cannot tell the two apart. Content-preserving policy rewrites flip up to 9.1% of verdicts above baseline jitter, and 18-43% of all observed flips occur on unambiguous cases under such rewrites, so existing safety scores conflate what the agent did with how the evaluator was prompted. Beyond the diagnosis, we contribute the Policy Invariance Score and the Judge Card reporting protocol, which expose an order-of-magnitude spread in judge reliability that is invisible to accuracy-only leaderboards. We release the protocol and code so that future agent-safety benchmarks can audit their own evaluators rather than trust them by default.

Related papers

• Credit Without Ground Truth: Auditing Step-Level Credit Assignment in LLM Agents Against Executed Replay [0.6768558752130311]
  We audit against policy-conditional ground truth from executed replay in a single-agent tool environment.<n>No step-level credit signals we audit show reliable incremental fidelity beyond marginal-matched shuffled control.<n>A confidence-only router recovers pivotal steps at chance level, but cuts judge cost by 13.1% per turn.
  arXiv  Detail & Related papers  (2026-08-20T08:04:00Z)
• Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees [16.73004625486562]
  Using LLMs as judges has become standard practice for evaluating model outputs at scale.<n>In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation.<n>We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts remains below a user-specified level.
  arXiv  Detail & Related papers  (2026-08-18T16:42:02Z)
• Adversarial Pragmatics for AI Safety Evaluation: A Benchmark for Instruction Conflict, Embedded Commands, and Policy Ambiguity [0.0]
  This paper introduces adversarial pragmatics as a benchmark for evaluating model behaviour under instruction conflict, embedded commands, quotation, scope ambiguity, deixis, indirect speech acts, and multi-turn agent transcripts.<n>The contribution is empirical and methodological: a linguistically controlled taxonomy, an 18-item seed benchmark with validator-enforced metadata, a 54-row local seed pilot, an expert-evaluation protocol distinguishing task success, policy compliance, safety risk, refusal outcome, and evaluator confidence, and metrics for judge validity, diagnostic ambiguity, and taxonomy drift.
  arXiv  Detail & Related papers  (2026-07-01T16:33:14Z)
• What Benchmarks Don't Measure: The Case for Evaluating Abstention Competence in Autonomous Agents [2.8340200313563773]
  Benchmarks for autonomous agents measure whether agents complete tasks, yet this framing is systematically blind to whether an agent should have proceeded at all.<n>Agents trained under human-feedback objectives develop a structural tendency to proceed even when they lack the inputs, evidence, or authorization to act safely.<n>We introduce a three-gap taxonomy of abstention-warranted scenarios, covering specification gaps where required information is absent, verification gaps where world state cannot be confirmed, and authority gaps where explicit authorization has not been given.
  arXiv  Detail & Related papers  (2026-06-01T23:52:56Z)
• Time to REFLECT: Can We Trust LLM Judges for Evidence-based Research Agents? [61.49434544687523]
  We introduce REFLECT, a meta-evaluation benchmark targeting fine-grained failure detection in agentic environments.<n> REFLECT defines a detailed taxonomy of process- and outcome-level failure modes, instantiated by performing controlled and localized interventions.<n>Our experiments show that even the best-performing models achieve overall accuracies below 55% across reasoning, tool-use, and report-quality failures.
  arXiv  Detail & Related papers  (2026-05-18T23:55:08Z)
• Rethinking Uncertainty in Segmentation: From Estimation to Decision [0.0]
  In medical image segmentation, uncertainty estimates are often reported but rarely used to guide decisions.<n>We formulate segmentation as a two-stage pipeline, estimation followed by decision, and show that optimizing uncertainty alone fails to capture most of the achievable safety gains.<n>Our results show that the best method and policy combination removes up to 80 percent of segmentation errors at only 25 percent pixel deferral.
  arXiv  Detail & Related papers  (2026-04-14T19:52:05Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• Gaming the Judge: Unfaithful Chain-of-Thought Can Undermine Agent Evaluation [76.5533899503582]
  Large language models (LLMs) are increasingly used as judges to evaluate agent performance.<n>We show this paradigm implicitly assumes that the agent's chain-of-thought (CoT) reasoning faithfully reflects both its internal reasoning and the underlying environment state.<n>We demonstrate that manipulated reasoning alone can inflate false positive rates of state-of-the-art VLM judges by up to 90% across 800 trajectories spanning diverse web tasks.
  arXiv  Detail & Related papers  (2026-01-21T06:07:43Z)
• The Stability Trap: Evaluating the Reliability of LLM-Based Instruction Adherence Auditing [1.5954459915735735]
  This study asks: To what extent does the instruction type of an Application Under Test (AUT) influence the stability of judge evaluations?<n>We introduce the Scoped Instruction Decomposition Framework to classify AUT instructions into Objective and Subjective types, isolating the factors that drive judge instability.<n>Our results reveal a Stability Trap'' characterized by a divergence between Verdict Stability and Reasoning Stability.
  arXiv  Detail & Related papers  (2026-01-16T21:15:13Z)
• Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations [49.84786015324238]
  Confidence estimation (CE) indicates how reliable the answers of large language models (LLMs) are, and can impact user trust and decision-making.<n>We present a comprehensive evaluation framework for CE that measures their confidence quality on three new aspects.<n>These include robustness of confidence against prompt perturbations, stability across semantic equivalent answers, and sensitivity to semantically different answers.
  arXiv  Detail & Related papers  (2026-01-12T23:16:50Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)
• Criticality and Safety Margins for Reinforcement Learning [53.10194953873209]
  We seek to define a criticality framework with both a quantifiable ground truth and a clear significance to users.<n>We introduce true criticality as the expected drop in reward when an agent deviates from its policy for n consecutive random actions.<n>We also introduce the concept of proxy criticality, a low-overhead metric that has a statistically monotonic relationship to true criticality.
  arXiv  Detail & Related papers  (2024-09-26T21:00:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.