A Layered Analysis of Disagreement And Answer Quality in Multi-Agent LLM Debate

• URL: http://arxiv.org/abs/2609.08016v1
• Date: Mon, 07 Sep 2026 21:57:44 GMT
• Title: A Layered Analysis of Disagreement And Answer Quality in Multi-Agent LLM Debate
• Abstract summary: Multi-agent debate is widely assumed to improve answer quality by surfacing genuine disagreement.<n>We evaluate 750 debates under three tones: friendly (seek common ground), neutral, and hostile.<n>For final answers we detect no quality gain: a bias-checked jury returns 299/299 ties, accuracy on a verifiable control task is unchanged, and a jury without the bias check had declared debate the winner 66% of the time.
• Score: 12.15064553632491
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-agent debate, in which several LLMs exchange arguments before answering, is widely assumed to improve answer quality by surfacing genuine disagreement. That mechanism is rarely checked. We introduce four measurements: (A) the agreement a debater reports; (B) whether its reply text actually pushes back; (C) whether the position persists once the eliciting instruction is removed; and (D) for open-weight models, the stance response in the debater's own token log-probabilities. We evaluate three-model committees debating open-ended GlobalOpinionQA across 750 debates under three tones: friendly (seek common ground), neutral, and hostile (stress-test every position). (A) Tone strongly reshapes reported agreement: full agreement differs by 50.4 percentage points between the friendly and hostile endpoints. (B) A judge that reads only the reply text, never the self-report or the condition, recovers the same pattern. (C) The dissent appears partly tied to the instruction that elicited it: labels revert toward agreement 23.1 points more often after deleting the hostile instruction than under a matched re-ask that keeps it; question-weighted inference is inconclusive on first-round turns alone (p=0.0625), significant pooling all rounds (p=0.016), and only 11/28 first-round reversions also appear in the reply text. (D) Opposing arguments weaken a debater's stance margin more consistently than they shift its direction. For final answers we detect no quality gain: a bias-checked jury returns 299/299 ties (ruling out only large differences), accuracy on a verifiable control task is unchanged, and a jury without the bias check had declared debate the winner 66% of the time -- an artifact of reading order. Taken together, LLM debate readily changes what agents say, but we find much weaker evidence that it changes what they persistently endorse or improves the quality of the final answer.

Related papers

• FramingQA: Does the Question Shape the Answer? Measuring the Compositional Framing Effect [69.8983512616053]
  Large language models often change their responses to subtle rephrasings that align with an implied stance by users.<n>This can leave users with advice tainted by how they happened to phrase a question rather than by the underlying facts.<n>We introduce FramingQA, a benchmark that measures the model sensitivity to question framing across law, medicine, finance, and robotic simulations.
  arXiv  Detail & Related papers  (2026-09-07T12:56:44Z)
• Heterogeneous LLM Debate Under Adversarial Peers: Honest Gains, Replacement Costs, and Resilience [1.3299507495084417]
  We measure how a heterogeneous peer changes the honest agents' revision behavior.<n>An honest peer sharply lowers harmful revision, and an adversarial one reverses it.<n>We also measure the effects when an adversarial same-family peer is already present.
  arXiv  Detail & Related papers  (2026-06-18T06:06:28Z)
• Collaborative Disagreement Resolution for Scalable Oversight [26.071237160687243]
  Disagreement resolution helps non-expert models identify the truth, achieving 62.1% judging accuracy compared to 49.2% for standard debate.<n>Our results provide encouraging empirical evidence for rethinking the scalable oversight protocol from adversarial persuasion to collaborative truth-seeking.
  arXiv  Detail & Related papers  (2026-06-02T12:55:49Z)
• The Deliberative Illusion: Diagnosing Factual Attrition and Stance Homogenization in Multi-Agent LLM Deliberation [68.98390038721963]
  We show that multi-agent discussion erases up to 72% of issue-critical facts.<n>This loss is consequential: retained evidence can reconstruct the issue misleadingly, final stances remain anchored in base-model priors, and a single malicious agent can inject misinformation into the shrinking shared context.<n>We call for evaluations that measure which facts, uncertainties, and legitimate disagreements survive interaction.
  arXiv  Detail & Related papers  (2026-06-02T02:15:40Z)
• MAIGO: Mitigating Lost-in-Conversation with History-Cleaned On-Policy Self-Distillation [51.32912774464992]
  Large language models often solve tasks from a fully specified prompt but degrade when the same requirements unfold over multiple turns.<n>We propose MAIGO, an on-policy self-distillation method that reduces this contamination using history-cleaned references from the model's own policy.<n>For middle turns, MAIGO removes prior assistant replies while preserving the user-visible sharded prefix; for answer turns, it distills from paired full-view references conditioned on the completed user-side dialogue.
  arXiv  Detail & Related papers  (2026-05-26T15:38:46Z)
• Debate Helps Weak Judges Reward Stronger Models [3.8081143468439147]
  We show that debate helps judge over a consultancy baseline when the critic's ability must exceed the judge's.<n>On the two non-responder pairings in our set, debate produces null effects, and judge verification rates drop by percentage points once a critic enters the transcript.<n>These findings suggest a cheaper primitive for trainingfree scalable oversight in verifiable domains.
  arXiv  Detail & Related papers  (2026-05-26T13:02:35Z)
• GRASP: Deterministic argument ranking in interaction graphs [65.66879897437157]
  We show that holistic judging suffers from substantial inter-model disagreement.<n>This instability arises from collapsing a debate's complex interaction structure into a single opaque score.<n>We propose GRASP, a deterministic framework that aggregates stable local interaction judgments into a global ranking.
  arXiv  Detail & Related papers  (2026-05-18T21:49:02Z)
• Measuring Opinion Bias and Sycophancy via LLM-based Persuasion [8.399156116912904]
  We propose a method for discovering the opinions an assistant holds on contested topics.<n>Direct probing asks for the model's opinion across five turns of escalating pressure from a simulated user.<n>Indirect probing never asks for an opinion and engages the model in argumentative debate, letting bias leak through how it concedes, resists, or counter-argues.
  arXiv  Detail & Related papers  (2026-04-23T11:34:06Z)
• Towards Scalable Oversight with Collaborative Multi-Agent Debate in Error Detection [81.52796950244705]
  Self-diagnosis is unreliable on complex tasks unless aided by reliable external feedback.<n>We introduce a new collaborative MAD protocol, termed ColMAD, that reframes MAD as a non-zero sum game.<n>We show that ColMAD significantly outperforms previous competitive MAD by 19%.
  arXiv  Detail & Related papers  (2025-10-23T19:46:00Z)
• Is the Top Still Spinning? Evaluating Subjectivity in Narrative Understanding [40.78154629252038]
  Forcing binary labels upon ambiguous claims lowers the reliability of evaluation.<n>We introduce LLM-generated edits of summaries as a method of providing a nuanced evaluation of claims.<n>We show that ARM produces a absolute 21% improvement in annotator agreement on claim faithfulness.
  arXiv  Detail & Related papers  (2025-04-01T19:08:24Z)
• When Persuasion Overrides Truth in Multi-Agent LLM Debates: Introducing a Confidence-Weighted Persuasion Override Rate (CW-POR) [0.46040036610482665]
  In many real-world scenarios, a single Large Language Model (LLM) may encounter contradictory claims-some accurate, others forcefully incorrect-and must judge which is true.<n>We investigate this risk in a single-turn, multi-agent debate framework: one LLM-based agent provides a factual answer from TruthfulQA, another vigorously defends a falsehood, and the same architecture serves as judge.<n>We introduce the Confidence-Weighted Persuasion Rate (CW-POR), which captures not only how often the judge is deceived but also how strongly it believes the incorrect choice.
  arXiv  Detail & Related papers  (2025-04-01T02:45:02Z)
• On scalable oversight with weak LLMs judging strong LLMs [67.8628575615614]
  We study debate, where two AI's compete to convince a judge; consultancy, where a single AI tries to convince a judge that asks questions. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models.
  arXiv  Detail & Related papers  (2024-07-05T16:29:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.