Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels

• URL: http://arxiv.org/abs/2605.29800v1
• Date: Thu, 28 May 2026 11:48:17 GMT
• Title: Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panels
• Abstract summary: LLM-as-a-judge panels aggregate votes from multiple models.<n>We quantify how far their reliability falls short of the independent-voting ideal.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: LLM-as-a-judge panels aggregate votes from multiple models, with the expectation that diverse models yield more reliable evaluations. We develop a framework to measure the true informational value of such panels and quantify how far their reliability falls short of the independent-voting ideal. Testing a panel of 9 frontier LLMs from 7 model families on three natural language inference datasets (each with 100 human annotations per item), we find that the 9 judges effectively provide only about 2 independent votes' worth of information. Roughly three-quarters of the panel's nominal independence is lost because the models make the same mistakes on the same items. The consequences are stark: the panel's actual accuracy falls 8-22 percentage points short of what independent voting would achieve, and the best single judge matches or outperforms the full panel across all conditions. Neither adding more judges nor using smarter aggregation algorithms helps -- established methods close at most 11% of this gap, even with access to the correct answers. We quantify these findings using the Kish effective sample size (n_eff) and a Condorcet null model, and show the deficit is robust across prompt variants, temperatures, chain-of-thought reasoning, and a pairwise preference task (RewardBench). The bottleneck is correlated judges, not the aggregation algorithm, implying that scaling up panels cannot substitute for genuinely independent evaluation.

Related papers

• Whose Gold? Annotator-Pool Disagreement Is Large at the Item Level, and Hidden by Small Leaderboards [0.0]
  We show that preference benchmarks are built by hiring annotators, and the identity of those annotators is treated as an implementation detail.<n>On the 2,885 MultiPref items where both pools are internally unanimous, so no tie-breaking convention is consulted at all.<n>Yet on both corpora the resulting model leaderboards are bit-identical: Kendall tau = 1.00 with zero of six models displaced.
  arXiv  Detail & Related papers  (2026-08-17T00:19:52Z)
• Forged Peer Judgments Mislead Multimodal LLM Judge Panels: Source-Blind Anchoring and Panel-Consensus Verification [9.710464466895521]
  Quoting independent visual judgments creates large anchoring gaps under both self and peer framing.<n>We introduce panel-consensus verification, which cross-checks a quote against independently collected blind votes.
  arXiv  Detail & Related papers  (2026-08-08T04:55:45Z)
• Blind to the Pivotal Vote: Aggregate Independence Metrics Miss Where Verification Actually Helps [9.710464466895521]
  We show that a majority-side replacement rule raises overall accuracy from 82.44% to 85.62% while invoking the signal on 16.2% of queries.<n>On HumanEval+/MBPP+, a majority-side replacement rule raises overall accuracy from 82.44% to 85.62% while invoking the signal on 16.2% of queries.
  arXiv  Detail & Related papers  (2026-08-07T08:14:46Z)
• Visual Credit Audit for Multimodal Spatial Reasoning [70.16915309526443]
  Visual Credit Audit separates two Estimands: whether the benchmark image gives the model's declared decision more support than text-only and blank controls, and whether the model responds to relation-specific visual evidence.<n>Applying labels yields dependence-credited correctness (D-CC)<n>Across four open MLLMs and two spatial benchmarks, 12.73-26.25% of decisions are correct yet uncredited.
  arXiv  Detail & Related papers  (2026-07-29T15:55:31Z)
• The Model as One Rater Among Several: Measuring Political Positions in Data-Sparse Regions with a Language-Model Panel [0.0]
  I describe the panel, an applicability rule that keeps a score of zero distinct from a blank, and a lens system that separates what an actor says from what it does.<n>I try to be plain about what the method can't do, including the human validation it still lacks, and I release the instrument and data in full.
  arXiv  Detail & Related papers  (2026-06-22T08:48:50Z)
• SLMJury: Can Small Language Models Judge as Well as Large Ones? [1.5990700377571574]
  We introduce SLMJury, a framework for evaluating small language models (SLMs) as judges across two paradigms.<n>We benchmark 16 SLM judges (0.6B-14B parameters) from four model families across ten benchmarks.<n> Reliable automated evaluation does not require large proprietary models, yet no single SLM dominates.
  arXiv  Detail & Related papers  (2026-06-05T19:38:15Z)
• CoEval: Ranking Language Models for Custom Tasks Without Labeled Data or Trustworthy Benchmarks [47.027290803102666]
  We present CoEval, an open framework that supplies a trustworthy, task-specific signal through ensemble self-evaluation.<n>A pool of models rotates through all three roles, teacher, student, and judge, to generate a fresh, contamination-free benchmark.
  arXiv  Detail & Related papers  (2026-06-02T13:41:43Z)
• Diversity Matters: Revisiting Test-Time Compute in Vision-Language Models [60.9932576020062]
  Test-time compute strategies have emerged as a lightweight approach to boost reasoning in large language models.<n>We propose Entropy-based TTC, which selects the most confident prediction based on predictive entropy.<n>We prove that ETTC outperforms majority voting under mild assumptions and empirically demonstrate that it consistently surpasses both voting and the best individual model.
  arXiv  Detail & Related papers  (2026-05-29T01:06:38Z)
• Visual Aesthetic Benchmark: Can Frontier Models Judge Beauty? [59.923111838399144]
  We introduce the Visual Aesthetic Benchmark (VAB), which casts aesthetic evaluation as comparative selection over candidate sets with matched subject matter.<n>VAB contains 400 tasks and 1,195 images across fine art, photography, and illustration, with labels derived from the consensus of 10 independent expert judges per task.<n>We find that the strongest system identifies both the best and the worst image correctly across three random permutations of the candidate order in only 26.5% of tasks, far below the 68.9% achieved by human experts.
  arXiv  Detail & Related papers  (2026-05-12T19:33:28Z)
• How Sensitive Are Safety Benchmarks to Judge Configuration Choices? [5.94231111588812]
  Safety benchmarks such as HarmBench rely on judges to classify model responses as harmful or safe.<n>We show that the combination of judge model and judge prompt is typically treated as a fixed implementation detail.<n>We find that prompt wording alone, holding the judge model fixed, shifts measured harmful-response rates by up to 24.2 percentage points.
  arXiv  Detail & Related papers  (2026-04-27T05:59:59Z)
• Judging the Judges: A Systematic Evaluation of Bias Mitigation Strategies in LLM-as-a-Judge Pipelines [0.0]
  We compare nine debiasing strategies across five judge models from four provider families.<n>Style bias is the dominant bias (0.76-0.92 across all models), far exceeding position bias (= 0.04)<n>The combined budget strategy significantly improves Claude Sonnet 4 by +11.2 pp (p 0.0001), with directionally positive trends for other models.
  arXiv  Detail & Related papers  (2026-04-25T07:18:30Z)
• MentorCollab: Selective Large-to-Small Inference-Time Guidance for Efficient Reasoning [85.05204262206296]
  Large reasoning models (LRMs) achieve strong performance by producing long chains of thought, but their inference costs are high.<n>Small language models (SLMs) are far more efficient, yet struggle on multi-step reasoning tasks.<n>We propose MentorCollab, an inference-time collaboration method in which an LRM selectively and sparsely guides an SLM, rather than taking over generation.
  arXiv  Detail & Related papers  (2026-02-05T04:58:16Z)
• Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models [55.94503936470247]
  Large-scale AI evaluation increasingly relies on aggregating binary judgments from $K$ annotators, including judges.<n>Most classical methods assume annotators are conditionally independent given the true label $Yin0,1$, an assumption often violated by LLM judges.<n>We study label aggregation through a hierarchy of dependence-aware models based on Ising graphical models and latent factors.
  arXiv  Detail & Related papers  (2026-01-29T21:26:50Z)
• Evaluative Fingerprints: Stable and Systematic Differences in LLM Evaluator Behavior [0.0]
  We find that judges are consistent, but not with each other; they are consistent with themselves.<n>Across 3,240 evaluations, inter-judge agreement is near-zero.<n>Averaging judges' scores produces a synthetic verdict that corresponds to no judge's actual values.
  arXiv  Detail & Related papers  (2026-01-08T17:02:22Z)
• Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers [63.99316853136304]
  Mirror-Critique is a framework that trains a verifier with informative critiques.<n>We deploy a small instruction-tuned model to synthesize high-quality critique data.<n>The resulting Mirror-Verifier is deployed to evaluate candidate solutions by generating multiple critiques per solution.
  arXiv  Detail & Related papers  (2025-09-27T06:50:24Z)
• TrustJudge: Inconsistencies of LLM-as-a-Judge and How to Alleviate Them [58.04324690859212]
  Large Language Models (LLMs) as automated evaluators (LLM-as-a-judge) has revealed critical inconsistencies in current evaluation frameworks.<n>We identify two fundamental types of inconsistencies: Score-Comparison Inconsistency and Pairwise Transitivity Inconsistency.<n>We propose TrustJudge, a probabilistic framework that addresses these limitations through two key innovations.
  arXiv  Detail & Related papers  (2025-09-25T13:04:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.