Statistical Scouting Finds Debate-Safe but Not Debate-Useful Cases: A Matched-Ceiling Study of Open-Weight LLM Reasoning Protocols

• URL: http://arxiv.org/abs/2605.09618v1
• Date: Sun, 10 May 2026 15:56:37 GMT
• Title: Statistical Scouting Finds Debate-Safe but Not Debate-Useful Cases: A Matched-Ceiling Study of Open-Weight LLM Reasoning Protocols
• Abstract summary: We evaluate greedy decoding, three-sample voting, and a two-agent critique-revise debate on MuSiQue and GSM8K.<n>On MuSiQue, an oracle selecting the correct protocol per example gains +14.0 and +13.7 pp over the best fixed one.<n>A vote-entropy threshold is the only controller that directionally beats the best fixed protocol on both models.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: When should a language model answer directly, sample and vote, or engage in multi-agent debate? Recent work shows voting often explains much of the gain attributed to debate, while selective-debate systems activate deliberation only on uncertain examples. We ask: under a matched ceiling on generated tokens (960 per example), how much per-example routing headroom exists, and how much is recoverable from cheap pre-deliberation signals? We evaluate greedy decoding, three-sample voting, and a two-agent critique-revise debate on MuSiQue and GSM8K using Llama 3.1 8B Instruct and Ministral 3 8B Instruct. On MuSiQue, an oracle selecting the correct protocol per example gains +14.0 and +13.7 pp over the best fixed one. The best fixed protocol is model- and dataset-dependent: each (model, dataset) cell has a different winner. This headroom is hard to recover from cheap ex-ante signals. A vote-entropy threshold is the only controller that directionally beats the best fixed protocol on both models (+1.3 and +1.7 pp), though individual paired-bootstrap CIs include zero. A joint analysis (meta-analysis +1.6 pp, p=0.125; Bayesian P(both>0)=0.59) is directionally consistent but not significant. Learned controllers (LR, GBT) do not outperform the threshold. The key finding is structural: vote entropy predicts where debate is safe, not where debate is needed. High entropy sharply reduces debate backfire, but 66% of debate-helpful examples (31/47) occur when voting is unanimous but wrong. A single-prompt self-critique probe on Llama flips the answer in 127/127 unanimous cases, yielding zero mutual information with the debate-helpful label; we cannot rule out a prompt-compliance artifact, but either interpretation disqualifies the probe as a router. Recovering the remaining headroom requires behavioral probes that avoid format-compliance confounds at the 8B scale.

Related papers

• A Layered Analysis of Disagreement And Answer Quality in Multi-Agent LLM Debate [12.15064553632491]
  Multi-agent debate is widely assumed to improve answer quality by surfacing genuine disagreement.<n>We evaluate 750 debates under three tones: friendly (seek common ground), neutral, and hostile.<n>For final answers we detect no quality gain: a bias-checked jury returns 299/299 ties, accuracy on a verifiable control task is unchanged, and a jury without the bias check had declared debate the winner 66% of the time.
  arXiv  Detail & Related papers  (2026-09-07T21:57:44Z)
• Marginal Fidelity Does Not Establish User Simulation in Demographic Synthetic Survey Panels: Response Contracts, Support Collapse and Conditioning Failure [51.736723807086385]
  Population-marginal agreement is evidence about an elicitation contract and an estimand obtainable without simulated respondents, not evidence of individual simulation.<n>On nine aligned model-battery pairs, a no-persona population-prevalence query averages 6.27 MAE versus 12.39 for committed panels and wins all nine comparisons.
  arXiv  Detail & Related papers  (2026-09-07T10:22:22Z)
• Who Flips? Self- and Cross-Model Counterarguments Reveal Answer Instability in LLMs [9.465306048183798]
  We introduce a controlled protocol for evaluating answer stability.<n>After a model answers a multiple-choice question correctly, we challenge the model with a coherent argument for an incorrect option.<n>We find that self-attribution consistently increases flip rates.
  arXiv  Detail & Related papers  (2026-06-14T20:45:30Z)
• The Chain Holds, the Answer Folds: Trace-Answer Dissociation in Reasoning Models Under Adversarial Pressure [8.86745721473138]
  Reasoning models are evaluated on single-turn benchmarks but deployed in multi-turn dialogue, where users push back on correct answers.<n>We call this unfaithful capitulation (UC) and isolate it with a $2times 2$ latent-versus-behavioral framework that flip-rate metrics and single-turn probes both miss.
  arXiv  Detail & Related papers  (2026-05-27T20:41:08Z)
• OpenDeepThink: Parallel Reasoning via Bradley-Terry Aggregation [53.88666485159289]
  We introduce OpenDeepThink, a population-based test-time compute framework that selects via pairwise Bradley-Terry comparison.<n>OpenDeepThink raises Gemini 3.1 Pro's effective Codeforces Elo by +405 points in eight sequential LLM-call rounds.<n>We release CF-73, a curated set of 73 expert-rated Codeforces problems with International Grandmaster annotation and 99% local-evaluation agreement against the official verdict.
  arXiv  Detail & Related papers  (2026-05-14T17:57:40Z)
• SURE-RAG: Sufficiency and Uncertainty-Aware Evidence Verification for Selective Retrieval-Augmented Generation [6.604874054866016]
  We present SURE-RAG, a transparent aggregation protocol built on the observation that evidence sufficiency is a set-level property.<n>A shared pair-level claim-evidence verifier produces local relation distributions, which SURE-RAG aggregates into interpretable answer-level signals.<n>We evaluate on HotpotQA-RAG v3, a controlled multi-hop benchmark, under an artifact-aware protocol.
  arXiv  Detail & Related papers  (2026-05-05T09:05:40Z)
• The Cost of Consensus: Isolated Self-Correction Prevails Over Unguided Homogeneous Multi-Agent Debate [5.51170856062205]
  We study teams of homogeneous agents across $R=3$ debate rounds on two high-difficulty benchmarks.<n>We decompose debate failure into three model-dependent pathways: sycophantic conformity, peer rationales, and consensus collapse.<n>Our results indicate that, within the 7-8B parameter class, homogeneous teams without structured roles do not benefit from unguided peer exchange.
  arXiv  Detail & Related papers  (2026-04-29T14:33:57Z)
• From Debate to Decision: Conformal Social Choice for Safe Multi-Agent Deliberation [12.294365308421606]
  We introduce Conformal Social Choice, a post-hoc decision layer that converts debate outputs into calibrated act-versus-escalate decisions.<n>A hierarchical action policy maps singleton sets to autonomous action and larger sets to human escalation.<n>Because the layer refuses to act on cases where debate is confidently wrong, the remaining conformal singletons reach 90.0--96.8% accuracy.
  arXiv  Detail & Related papers  (2026-04-09T00:15:20Z)
• Do Large Language Models Get Caught in Hofstadter-Mobius Loops? [0.0]
  This paper argues that modern RLHF-trained language models are subject to a structurally analogous contradiction.<n>The training process simultaneously rewards compliance with user preferences and suspicion toward user intent.<n>The resulting behavioral profile is consistent with what Clarke termed a Hofstadter-Mobius loop.
  arXiv  Detail & Related papers  (2026-03-10T20:43:37Z)
• Build, Borrow, or Just Fine-Tune? A Political Scientist's Guide to Choosing NLP Models [0.0]
  Political scientists increasingly face a consequential choice when adopting natural language processing tools.<n>Each approach occupies a different point on the spectrum of performance, cost, and required expertise.<n>This paper provides empirical guidance on how to navigate this trade-off.
  arXiv  Detail & Related papers  (2026-03-10T12:42:12Z)
• TARG: Training-Free Adaptive Retrieval Gating for Efficient RAG [46.122203287541005]
  Training-free Adaptive Retrieval Gating (TARG) is a single-shot policy that decides when to retrieve using only a short, no-latency draft from the base model.<n>On NQ-Open, TriviaQA, and PopQA, TARG consistently shifts the accuracy-efficiency frontier.
  arXiv  Detail & Related papers  (2025-11-12T23:09:52Z)
• DISCO: Diversifying Sample Condensation for Efficient Model Evaluation [59.01400190971061]
  Costly evaluation reduces inclusivity, slows the cycle of innovation, and worsens environmental impact.<n>We argue that promoting diversity among samples is not essential; what matters is to select samples thatmaximise diversity in model responses.<n>Our method, $textbfDiversifying Sample Condensation (DISCO)$, selects the top-k samples with the greatest model disagreements.
  arXiv  Detail & Related papers  (2025-10-09T08:53:59Z)
• Critique to Verify: Accurate and Honest Test-Time Scaling with RL-Trained Verifiers [63.99316853136304]
  Mirror-Critique is a framework that trains a verifier with informative critiques.<n>We deploy a small instruction-tuned model to synthesize high-quality critique data.<n>The resulting Mirror-Verifier is deployed to evaluate candidate solutions by generating multiple critiques per solution.
  arXiv  Detail & Related papers  (2025-09-27T06:50:24Z)
• DCR: Divide-and-Conquer Reasoning for Multi-choice Question Answering with LLMs [9.561022942046279]
  We propose Divide and Conquer Reasoning (DCR) to enhance the reasoning capability of large language models (LLMs) We first categorize questions into two subsets based on confidence score ($mathcalCS$), which is estimated by statistical frequency of generated answers. In particular, we first categorize questions into two subsets based on confidence score ($mathcalCS$), which is estimated by statistical frequency of generated answers.
  arXiv  Detail & Related papers  (2024-01-10T14:38:46Z)
• Large Language Models are not Fair Evaluators [60.27164804083752]
  We find that the quality ranking of candidate responses can be easily hacked by altering their order of appearance in the context. This manipulation allows us to skew the evaluation result, making one model appear considerably superior to the other. We propose a framework with three simple yet effective strategies to mitigate this issue.
  arXiv  Detail & Related papers  (2023-05-29T07:41:03Z)
• Global Constraints with Prompting for Zero-Shot Event Argument Classification [49.84347224233628]
  We propose to use global constraints with prompting to tackle event argument classification without any annotation and task-specific training. A pre-trained language model scores the new passages, making the initial prediction. Our novel prompt templates can easily adapt to all events and argument types without manual effort.
  arXiv  Detail & Related papers  (2023-02-09T06:39:29Z)
• WR-ONE2SET: Towards Well-Calibrated Keyphrase Generation [57.11538133231843]
  Keyphrase generation aims to automatically generate short phrases summarizing an input document. The recently emerged ONE2SET paradigm generates keyphrases as a set and has achieved competitive performance. We propose WR-ONE2SET which extends ONE2SET with an adaptive instance-level cost Weighting strategy and a target Re-assignment mechanism.
  arXiv  Detail & Related papers  (2022-11-13T09:56:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.