Multiagent Protocols with Aggregated Confidence Signals

• URL: http://arxiv.org/abs/2606.13591v1
• Date: Thu, 11 Jun 2026 17:12:11 GMT
• Title: Multiagent Protocols with Aggregated Confidence Signals
• Abstract summary: Confidence is used for reliability, oversight, and a range of downstream decision tasks in Natural Language Processing (NLP)<n>Prior work uses confidence within multiagent debate (MAD) to weight messages, trigger debate, or calibrate individual agents, but it never aggregates these into a single confidence for the system itself.<n>We introduce three protocols that produce a final answer along with a single aggregated confidence by first transforming raw confidence signals to make them comparable across models, then combining them via soft voting or a probability fusion.
• Score: 7.128020779227392
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Confidence is used for reliability, oversight, and a range of downstream decision tasks in Natural Language Processing (NLP), yet no existing method produces or evaluates a confidence for the output of a multiagent system. Prior work uses confidence within multiagent debate (MAD) to weight messages, trigger debate, or calibrate individual agents, but it never aggregates these into a single confidence for the system itself. We introduce three protocols that produce a final answer along with a single aggregated confidence by first transforming raw confidence signals to make them comparable across models, then combining them via soft voting or a probability fusion we call Bayesian fusion. This aggregated confidence is substantially more discriminative (AUARC) than that of the best single agent or the standard debate baselines, while correctness (F1-score) stays stable and recovers the losses MAD incurs on more ambiguous tasks. Analyzing two estimators, sequence probability and self-report, alongside parametric and non-parametric calibrators, we find that calibration improves F1 for both estimators while AUARC is less reliant on it. We evaluate six homogeneous and heterogeneous debating pairs per benchmark, across five benchmarks and four task types, spanning a range of model capabilities and sizes.

Related papers

• Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models [7.714704554667688]
  Large Language Models (LLMs) increasingly support decision-making in high-stakes domains.<n>LLMs often hallucinate and express confidence that is misaligned with factual correctness.<n>We study claim-level confidence calibration as a decision-relevant uncertainty signal.
  arXiv  Detail & Related papers  (2026-08-23T16:09:54Z)
• DirEAG: Dirichlet Evidence Aggregation for Calibrating Verbalized Confidence in Mathematical Reasoning [2.867517731896504]
  Black-box verbalized confidence is difficult to calibrate in large language models.<n>We propose DirEAG, a Dirichlet Evidence Aggregation method that converts each elicited answer-confidence observation into calibrated soft evidence.
  arXiv  Detail & Related papers  (2026-08-21T03:48:25Z)
• MARGIN: Runtime Confidence Calibration for Multi-Agent Foundation Model Coordination [0.0]
  We present MARGIN (Multi-Agent Grading via Incremental Normalisation), an online calibration method that learns per-agent, per-band calibration factors from the task stream itself.<n>Across 18 foundation models, 8 benchmarks, and over 44,000 observations, MARGIN achieves 3-6x lower calibration error than the best design-time baseline under distribution shift.
  arXiv  Detail & Related papers  (2026-05-21T18:25:05Z)
• Process Supervision of Confidence Margin for Calibrated LLM Reasoning [52.373121066425455]
  Scaling test-time computation with reinforcement learning (RL) has emerged as a reliable path to improve large language models (LLM) reasoning ability.<n>Yet, outcome-based reward often incentivizes models to be overconfident, leading to hallucinations, unreliable confidence-based control, and unnecessary compute allocation.<n>We introduce Reinforcement Learning with Confidence Margin (textbfRLCM), a calibration-aware RL framework that jointly optimize correctness and confidence reliability.
  arXiv  Detail & Related papers  (2026-04-25T14:40:13Z)
• Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Models [11.878003218072765]
  Multimodal Large Language Models (MLLMs) have demonstrated exceptional capabilities in various perception and reasoning tasks.<n>We propose a monotone confidence fusion framework to merge dual-channel signals and cross-channel consistency to estimate correctness.<n>Experiments on diverse open-source and closed-source MLLMs show that our method consistently yields more reliable confidence estimates.
  arXiv  Detail & Related papers  (2026-04-19T06:07:39Z)
• Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimation [47.91529693614168]
  Existing methods are predominantly answer-first, producing confidence only after generating an answer.<n>We study a confidence-first paradigm, where the model outputs its confidence before answering, interpreting this score as the model's probability of answering the question correctly.<n>We propose CoCA, a reinforcement learning framework that jointly optimize confidence calibration and answer accuracy via segmented credit assignment.
  arXiv  Detail & Related papers  (2026-03-06T04:03:13Z)
• On Calibration of Large Language Models: From Response To Capability [66.59139960234326]
  Large language models (LLMs) are widely deployed as general-purpose problem solvers.<n>We introduce capability calibration, which targets the model's expected accuracy on a query.<n>Our results demonstrate that capability-calibrated confidence improves pass@$k$ prediction and inference budget allocation.
  arXiv  Detail & Related papers  (2026-02-14T01:07:45Z)
• Confidence Estimation for LLMs in Multi-turn Interactions [48.081802290688394]
  This work presents the first systematic study of confidence estimation in multi-turn interactions.<n>We establish a formal evaluation framework grounded in two key desideratas: per-turn calibration and monotonicity of confidence.<n>Our work provides a foundational methodology for developing more reliable and trustworthy conversational agents.
  arXiv  Detail & Related papers  (2026-01-05T14:58:04Z)
• BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents [58.05949210993854]
  We investigate whether search agents have the ability to communicate their own confidence through verbalized confidence scores after long sequences of actions.<n>We propose Test-Time Scaling (TTS) methods that use confidence scores to determine answer quality, encourage the model to try again until reaching a satisfactory confidence level.
  arXiv  Detail & Related papers  (2025-10-27T15:58:51Z)
• ConfTuner: Training Large Language Models to Express Their Confidence Verbally [58.63318088243125]
  Large Language Models (LLMs) are increasingly deployed in high-stakes domains such as science, law, and healthcare.<n>LLMs are often observed to generate incorrect answers with high confidence, a phenomenon known as "overconfidence"
  arXiv  Detail & Related papers  (2025-08-26T09:25:32Z)
• SteerConf: Steering LLMs for Confidence Elicitation [11.872504642312705]
  Large Language Models (LLMs) exhibit impressive performance across diverse domains but often suffer from overconfidence.<n>We propose SteerConf, a novel framework that systematically steers LLMs' confidence scores to improve their calibration and reliability.
  arXiv  Detail & Related papers  (2025-03-04T18:40:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.