When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability

• URL: http://arxiv.org/abs/2607.08535v1
• Date: Thu, 09 Jul 2026 14:31:05 GMT
• Title: When the Judge Changes, So Does the Measurement: Auditing LLM-as-Judge Reliability
• Abstract summary: An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed.<n>We compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs.<n>The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax releases do not.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: An LLM-as-judge score can move even when the candidate responses stay fixed, simply because the evaluator has changed. We treat this evaluator-replacement ambiguity as a measurement-validity problem. Across four judgment datasets, we compare two upgrade paths available in practice: scaling Qwen3 dense judges from 1.7B to 32B parameters and moving across MiniMax M2-M2.7 released APIs. The main pattern is that judge upgrades are not interchangeable: only Qwen3 1.7B to 4B gives a robust adjacent gain, while MiniMax adjacent releases do not. Stronger judges reduce but do not remove position and verbosity bias. Repeated-sample juries add little when errors are correlated. Structured debate can move decisions substantially, but without parser and fallback logs those shifts cannot be attributed to deliberation. We argue that LLM-as-judge reports should include dataset slices, bias probes, error-dependence estimates, and protocol audit trails.

Related papers

• LLM-as-a-Judge Is Not an Oracle: Why Self-Improving Agents Need Deterministic Guardrails [0.0]
  Self-improving agent pipelines have a problem at their center.<n>An evaluation signal failed in eleven classes: judge bias, harness and metric failures, ground-truth errors, and reward hacking.<n>We describe PROCTOR, a Teacher-Student loop in which a stateful orchestrator holds all tool access, stateless subagents diagnose failures and draft mutations they cannot apply.
  arXiv  Detail & Related papers  (2026-09-02T07:54:23Z)
• Evidence Lock Before Commitment: A Frozen Interface Degrades LLM-as-Judge Evaluation [0.6768558752130311]
  This workflow assumes that the intermediate record preserves the information needed for a later verdict.<n>For reasoning-capable models, visible field order does not reveal internal decision order, so we test an observable alternative.<n>We compare standard pairwise judging, structured one-call judging, two-call evidence locking, and three-call pointwise locking with Claude Sonnet 4.5 and GPT-5.
  arXiv  Detail & Related papers  (2026-08-05T19:23:56Z)
• Turning Bias into Bugs: Bandit-Guided Style Manipulation Attacks on LLM Judges [65.92623263645139]
  We introduce BITE, a black-box adversarial framework that learns semantics-preserving edits to mislead an LLM judge.<n>BITE achieves an attack success rate exceeding 65% and raises scores by 1-2 points on a 9-point scale.<n>Our findings expose a fundamental weakness in the LLM-as-a-judge paradigm and motivate robust, attack-aware evaluation.
  arXiv  Detail & Related papers  (2026-05-24T05:24:09Z)
• C2-Faith: Benchmarking LLM Judges for Causal and Coverage Faithfulness in Chain-of-Thought Reasoning [0.6138671548064355]
  Large language models (LLMs) are increasingly used as judges of chain-of-thought (CoT) reasoning.<n>We introduce C2-Faith, a benchmark that targets two complementary dimensions of faithfulness: causality (does each step logically follow from prior context?) and coverage.<n>We evaluate three frontier judges under three tasks: binary causal detection, causal step localization, and coverage scoring.
  arXiv  Detail & Related papers  (2026-03-05T13:36:47Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• VideoJudge: Bootstrapping Enables Scalable Supervision of MLLM-as-a-Judge for Video Understanding [57.15309719147799]
  We introduce VideoJudge, a 3B and 7B-sized MLLM judge specialized to evaluate outputs from video understanding models.<n>To train VideoJudge, our recipe builds on the interplay between a generator and an evaluator.<n>Across three out of four meta-evaluation benchmarks, VideoJudge-7B outperforms larger MLLM judge baselines.
  arXiv  Detail & Related papers  (2025-09-25T19:22:57Z)
• No-Knowledge Alarms for Misaligned LLMs-as-Judges [0.0]
  We develop no-knowledge alarms for misaligned LLM judges.<n>The alarms can detect, with no false positives, that at least one member or more of an ensemble of judges are violating a user specified grading ability requirement.
  arXiv  Detail & Related papers  (2025-09-10T13:46:40Z)
• JudgeBench: A Benchmark for Evaluating LLM-based Judges [61.048125269475854]
  JudgeBench is a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding.<n>Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks.
  arXiv  Detail & Related papers  (2024-10-16T17:58:19Z)
• JudgeLM: Fine-tuned Large Language Models are Scalable Judges [48.053949045598344]
  evaluating Large Language Models (LLMs) in open-ended scenarios is challenging because existing benchmarks and metrics can not measure them comprehensively.<n>We propose to fine-tune LLMs as scalable judges (JudgeLM) to evaluate LLMs efficiently and effectively in open-ended benchmarks.<n>We train JudgeLM at different scales from 7B, 13B, to 33B parameters, and conduct a systematic analysis of its capabilities and behaviors.
  arXiv  Detail & Related papers  (2023-10-26T17:48:58Z)
• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena [76.21004582932268]
  We examine the usage and limitations of LLM-as-a-judge, including position, verbosity, and self-enhancement biases. We then verify the agreement between LLM judges and human preferences by introducing two benchmarks: MT-bench, a multi-turn question set; and Arena, a crowdsourced battle platform.
  arXiv  Detail & Related papers  (2023-06-09T05:55:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.