A Theory of Post-hoc Debate Judgement

• URL: http://arxiv.org/abs/2608.19002v1
• Date: Wed, 19 Aug 2026 15:03:44 GMT
• Title: A Theory of Post-hoc Debate Judgement
• Abstract summary: We develop and test a novel theory of debate judgement applicable to all settings where agents engage in debates.<n>We identify a number of formal properties that debate judgement may be required to satisfy in general, as concerns, robustness, groundedness and explainability.<n>We show that the two methods give similar accuracy performances but the former may lack formal guarantees.
• Score: 29.17703958536647
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Debates have recently emerged as a useful methodology for agentic AI to improve performance as well as to aid explainability and user engagement. For example, LLM-empowered agents may debate internally (with themselves) and/or externally (with other agents). In many settings where debates are used, debates' outcomes and resulting outputs are determined post-hoc by external judges, often LLMs. In this paper we develop and test a novel theory of debate judgement applicable to all settings where agents engage in debates by providing pros and cons for their opinions therein. Specifically, we identify a number of formal properties that debate judgement may be required to satisfy in general, as concerns reproducibility, robustness, groundedness and explainability. Then, we explore their satisfaction formally and/or experimentally, for claim verification settings, for two specific alternative debate judgement methods: variants of the LLMs as a judge idea and formal semantics drawn from computational argumentation. We show that the two methods give similar accuracy performances but the former may lack formal guarantees that the latter brings. Overall, our study indicates argumentation semantics as an ideal candidate for principled judges in debate-driven AI.

Related papers

• Knowledge Divergence and the Value of Debate for Scalable Oversight [0.0]
  Debate and reinforcement learning from AI feedback are proposed methods for scalable oversight of advanced AI systems.<n>We analyze this by parameterizing debate's value through the geometry of knowledge divergence between debating models.<n>We offer the first formal connection between debate and RLAIF, a geometric foundation for understanding when adversarial oversight protocols are justified.
  arXiv  Detail & Related papers  (2026-03-05T15:36:08Z)
• AI-Assisted Moot Courts: Simulating Justice-Specific Questioning in Oral Arguments [7.808898285349819]
  We examine whether AI models can effectively simulate justice-specific questioning for moot court-style training.<n>We construct and evaluate both prompt-based and agentic oral argument simulators.<n>We find that simulated questions are often perceived as realistic by human annotators and achieve high recall of ground truth substantive legal issues.
  arXiv  Detail & Related papers  (2026-03-05T01:45:28Z)
• R-Debater: Retrieval-Augmented Debate Generation through Argumentative Memory [18.007379464461312]
  We present R-Debater, an agentic framework for generating multi-turn debates built on argumentative memory.<n>R-Debater integrates a debate knowledge base for retrieving case-like evidence and prior debate moves with a role-based agent that composes coherent utterances across turns.
  arXiv  Detail & Related papers  (2025-12-31T07:33:12Z)
• Debating Truth: Debate-driven Claim Verification with Multiple Large Language Model Agents [13.626715532559079]
  We propose DebateCV, the first claim verification framework that adopts a debate-driven methodology using multiple LLM agents.<n>In our framework, two Debaters take opposing stances on a claim and engage in multi-round argumentation, while a Moderator evaluates the arguments and renders a verdict with justifications.<n> Experimental results show that our method outperforms existing claim verification methods under varying levels of evidence quality.
  arXiv  Detail & Related papers  (2025-07-25T09:19:25Z)
• Arbiters of Ambivalence: Challenges of Using LLMs in No-Consensus Tasks [52.098988739649705]
  This study examines the biases and limitations of LLMs in three roles: answer generator, judge, and debater.<n>We develop a no-consensus'' benchmark by curating examples that encompass a variety of a priori ambivalent scenarios.<n>Our results show that while LLMs can provide nuanced assessments when generating open-ended answers, they tend to take a stance on no-consensus topics when employed as judges or debaters.
  arXiv  Detail & Related papers  (2025-05-28T01:31:54Z)
• Debating for Better Reasoning: An Unsupervised Multimodal Approach [56.74157117060815]
  We extend the debate paradigm to a multimodal setting, exploring its potential for weaker models to supervise and enhance the performance of stronger models.<n>We focus on visual question answering (VQA), where two "sighted" expert vision-language models debate an answer, while a "blind" (text-only) judge adjudicates based solely on the quality of the arguments.<n>In our framework, the experts defend only answers aligned with their beliefs, thereby obviating the need for explicit role-playing and concentrating the debate on instances of expert disagreement.
  arXiv  Detail & Related papers  (2025-05-20T17:18:17Z)
• On scalable oversight with weak LLMs judging strong LLMs [67.8628575615614]
  We study debate, where two AI's compete to convince a judge; consultancy, where a single AI tries to convince a judge that asks questions. We use large language models (LLMs) as both AI agents and as stand-ins for human judges, taking the judge models to be weaker than agent models.
  arXiv  Detail & Related papers  (2024-07-05T16:29:15Z)
• Debatrix: Multi-dimensional Debate Judge with Iterative Chronological Analysis Based on LLM [51.43102092480804]
  Debatrix is an automated debate judge based on Large Language Models (LLMs) To align with real-world debate scenarios, we introduced the PanelBench benchmark, comparing our system's performance to actual debate outcomes. The findings indicate a notable enhancement over directly using LLMs for debate evaluation.
  arXiv  Detail & Related papers  (2024-03-12T18:19:47Z)
• Reward Design for Justifiable Sequential Decision-Making [12.284934135116515]
  We propose the use of a debate-based reward model for reinforcement learning agents. We show that augmenting the reward with the feedback signal generated by the debate-based reward model yields policies highly favored by the judge.
  arXiv  Detail & Related papers  (2024-02-24T14:29:30Z)
• A Unifying Framework for Learning Argumentation Semantics [47.84663434179473]
  We present a novel framework, which uses an Inductive Logic Programming approach to learn the acceptability semantics for several abstract and structured argumentation frameworks in an interpretable way.<n>Our framework outperforms existing argumentation solvers, thus opening up new future research directions in the area of formal argumentation and human-machine dialogues.
  arXiv  Detail & Related papers  (2023-10-18T20:18:05Z)
• Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate [85.3444184685235]
  We propose a Multi-Agent Debate (MAD) framework, in which multiple agents express their arguments in the state of "tit for tat" and a judge manages the debate process to obtain a final solution. Our framework encourages divergent thinking in LLMs which would be helpful for tasks that require deep levels of contemplation.
  arXiv  Detail & Related papers  (2023-05-30T15:25:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.