Debate Helps Weak Judges Reward Stronger Models

• URL: http://arxiv.org/abs/2605.27483v1
• Date: Tue, 26 May 2026 13:02:35 GMT
• Title: Debate Helps Weak Judges Reward Stronger Models
• Abstract summary: We show that debate helps judge over a consultancy baseline when the critic's ability must exceed the judge's.<n>On the two non-responder pairings in our set, debate produces null effects, and judge verification rates drop by percentage points once a critic enters the transcript.<n>These findings suggest a cheaper primitive for trainingfree scalable oversight in verifiable domains.
• Score: 3.8081143468439147
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite theoretical promise, debate as a scalable oversight protocol has produced mixed empirical results: gains in some settings, and null effects in others, especially when the judge does not have information hidden from it. We study proposer-critic debate in a stronger-debater/weaker-judge setting on programmatically verifiable code and logic tasks. Debate helps the judge over a consultancy baseline when the critic provides a usable advantage: the critic's classification ability must exceed the judge's, and the judge must treat critic speeches as claims to verify rather than testimony to summarize. On the three of five pairings where the condition holds, proposer-critic debate's gains are statistically significant over consultancy, and these pairings are the most capable model pairings. On the two non-responder pairings in our set, debate produces null effects, and judge verification rates drop by tens of percentage points once a critic enters the transcript. In these cases the critic's binary-classification ability and the judge's are within noise of each other, and the critic's disagreement is parsed as testimony rather than a claim to check. Ablating rebuttal rounds from debate produces no measurable change in judge performance: a single independent critique recovers the bulk of debate's benefit at lower inference cost. These findings suggest a cheaper primitive for training-free scalable oversight in verifiable domains (answer, critique, judge) and a pre-deployment audit (does the critic beat the judge, and will the judge verify it?) that predicts when debate will help.

Related papers

• A Layered Analysis of Disagreement And Answer Quality in Multi-Agent LLM Debate [12.15064553632491]
  Multi-agent debate is widely assumed to improve answer quality by surfacing genuine disagreement.<n>We evaluate 750 debates under three tones: friendly (seek common ground), neutral, and hostile.<n>For final answers we detect no quality gain: a bias-checked jury returns 299/299 ties, accuracy on a verifiable control task is unchanged, and a jury without the bias check had declared debate the winner 66% of the time.
  arXiv  Detail & Related papers  (2026-09-07T21:57:44Z)
• A Theory of Post-hoc Debate Judgement [29.17703958536647]
  We develop and test a novel theory of debate judgement applicable to all settings where agents engage in debates.<n>We identify a number of formal properties that debate judgement may be required to satisfy in general, as concerns, robustness, groundedness and explainability.<n>We show that the two methods give similar accuracy performances but the former may lack formal guarantees.
  arXiv  Detail & Related papers  (2026-08-19T15:03:44Z)
• Sharding Prevents LLM Oversight Failures and Adversarial Exploitation [45.380649793616705]
  Sharding partitions requirements into smaller groups, assigns each group to a separate call, and aggregates the verdicts.<n>We find that a sharded weaker judge can outperform a more capable holistic judge and match that judge even when the latter receives the panel's full budget.
  arXiv  Detail & Related papers  (2026-08-05T18:21:21Z)
• How Long Reasoning Chains Influence LLMs' Judgment of Answer Factuality [9.19183567561999]
  Large language models (LLMs) has been widely adopted as a scalable surrogate for human evaluation, yet such judges remain imperfect and susceptible to surface-level biases.<n>With the rise of reasoning-capable models, exposing a generator's reasoning content to the judge provides richer information and is a natural candidate for improving judgment accuracy.<n>We find that weak judges are easily swayed by reasoning presence, while strong judges can partially leverage reasoning as informative evidence.
  arXiv  Detail & Related papers  (2026-04-08T07:21:18Z)
• Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimization [44.252712888022835]
  Large language model (LLM)-based judges are widely adopted for automated evaluation and reward modeling.<n>We propose JudgeBiasBench, a benchmark for systematically quantifying biases in LLM-based judges.<n>We conduct experiments across both generative and discriminative judges, revealing that current judges exhibit significant and diverse bias patterns.
  arXiv  Detail & Related papers  (2026-03-09T08:32:21Z)
• AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments [7.808898285349819]
  We examine whether AI models can effectively simulate justice-specific questioning for moot court-style training.<n>We construct and evaluate both prompt-based and agentic oral argument simulators.<n>We find that simulated questions are often perceived as realistic by human annotators and achieve high recall of ground truth substantive legal issues.
  arXiv  Detail & Related papers  (2026-03-05T01:45:28Z)
• A Coin Flip for Safety: LLM Judges Fail to Reliably Measure Adversarial Robustness [57.510025257780306]
  We show that existing validation protocols fail to account for substantial distribution shifts inherent to red-teaming.<n>We propose ReliableBench, a benchmark of behaviors that remain more consistently judgeable, and JudgeStressTest, a dataset designed to expose judge failures.
  arXiv  Detail & Related papers  (2026-02-04T15:13:35Z)
• AI Debaters are More Persuasive when Arguing in Alignment with Their Own Beliefs [0.13525723298325706]
  We apply debate to subjective questions and explicitly measure large language models' prior beliefs before experiments.<n>We implement and compare two debate protocols, sequential and simultaneous, to evaluate potential systematic biases.<n>Our main findings show that models tend to prefer defending stances aligned with the judge persona rather than their prior beliefs.
  arXiv  Detail & Related papers  (2025-10-15T05:02:13Z)
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards [72.44810390478229]
  CompassJudger-2 is a novel generalist judge model that overcomes limitations via a task-driven, multi-domain data curation strategy.<n> CompassJudger-2 achieves superior results across multiple judge and reward benchmarks.
  arXiv  Detail & Related papers  (2025-07-12T01:34:24Z)
• AI Debate Aids Assessment of Controversial Claims [73.8907110799657]
  We study whether AI debate can guide biased judges toward the truth by having two AI systems debate opposing sides of controversial factuality claims.<n>In Study I, debate consistently improves human judgment accuracy and confidence calibration, outperforming consultancy.<n>In Study II, AI judges with human-like personas achieve even higher accuracy (78.5%) than human judges (70.1%) and default AI judges without personas (69.8%)<n>These findings highlight AI debate as a promising path toward scalable, bias-resilient oversight in contested domains.
  arXiv  Detail & Related papers  (2025-06-02T19:01:53Z)
• JudgeBench: A Benchmark for Evaluating LLM-based Judges [61.048125269475854]
  JudgeBench is a benchmark for evaluating LLM-based judges on challenging response pairs spanning knowledge, reasoning, math, and coding.<n>Our comprehensive evaluation on a collection of prompted judges, fine-tuned judges, multi-agent judges, and reward models shows that JudgeBench poses a significantly greater challenge than previous benchmarks.
  arXiv  Detail & Related papers  (2024-10-16T17:58:19Z)
• On scalable oversight with weak LLMs judging strong LLMs [67.8628575615614]
  We study debate, where two AI's compete to convince a judge; consultancy, where a single AI tries to convince a judge that asks questions. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models.
  arXiv  Detail & Related papers  (2024-07-05T16:29:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.