When Agents Disagree: Bayesian Backward Reasoning as a Label-Free Anchor for Multi-Agent Collective Decision-Making

• URL: http://arxiv.org/abs/2609.11709v1
• Date: Thu, 10 Sep 2026 15:27:58 GMT
• Title: When Agents Disagree: Bayesian Backward Reasoning as a Label-Free Anchor for Multi-Agent Collective Decision-Making
• Abstract summary: LLM judges, voting rules, and electoral rules rely on forward reasoning to map evidence to labels in one direction.<n>We construct a reverse posterior for each instance through Bayesian backward reasoning from an explicit likelihood.<n>Because estimates from different factorizations may tend to share the same error less often, we use Jensen-Shannon divergence to rank agents by cross-path consistency.
• Score: 5.335515762403034
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: When multiple LLM agents yield conflicting answers, the decision-making process dictates whether agent diversity improves performance or merely compounds shared errors. Existing collective decision-making methods, including voting, electoral rules, and LLM judges, rely on forward reasoning: they map evidence to labels in one direction. Although these methods can combine diverse forward traces, they still aggregate estimates that share this evidence-to-label factorization and can inherit correlated errors within the forward pool. We therefore construct a reverse posterior for each instance through Bayesian backward reasoning from an explicit likelihood. The forward and reverse posteriors provide differently factorized approximations of the underlying posterior. Because estimates from different factorizations may tend to share the same error less often, we use Jensen-Shannon divergence to rank agents by cross-path consistency. This cross-path consistency signal underlies three strategies: hard selection (MinJS), soft reweighting (FwdJS), and log-linear fusion (LogLin). Evaluated on DDXPlus across five LLM backbones, our proposed strategies show consistent improvements: MinJS outperforms random selection across all backbones, FwdJS generally improves over the strongest baseline, and LogLin achieves the best performance among the evaluated methods, with its largest gains on the subset where the agents disagree. Despite its weaker standalone accuracy, the reverse posterior serves as a more useful anchor than forward-only alternatives, providing complementary information for collective decision-making. When labeled data are available, a lightweight two-stage calibration can further refine the reverse anchor and improve aggregation performance.

Related papers

• Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents [0.0]
  We study whether three common families of single-turn UQ methods transfer to an interactive trajectory setting.<n>We find that transfer is often useful but uneven. Token-probability scores are highly sensitive to the choice of aggregator used across turns.<n>Black-box self-consistency is often the strongest UQ family, with trajectory-equivalence and action-set consistency typically ranking highest among its variants.
  arXiv  Detail & Related papers  (2026-08-12T01:39:28Z)
• Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present? [3.77661940418406]
  Methods for query expansion, hierarchical and cross-document summarization, graph-based expansion, per-query routing, rank fusion, and corrective re-retrieval are studied.<n>We show that a strong cross-encoder reranker accounts for most of the pipeline's quality.<n>The remaining reranking and pool-expansion methods in common use, among them hierarchical summarization, graph expansion, routing, and rank fusion, give no reliable gain once that reranker is present.
  arXiv  Detail & Related papers  (2026-06-14T20:50:41Z)
• DISA: Offline Importance Sampling for Distribution-Matching LLM-RL [56.9445657766829]
  We introduce DISA, which moves this calibration problem outside the RL loop.<n> DISA draws proposal trajectories offline, estimates the partition function via importance sampling, and freezes the resulting partition-function estimate.<n>On two open-weight backbones across six math and three code benchmarks, DISA matches or exceeds the online-coupled distribution-matching baseline Flow.
  arXiv  Detail & Related papers  (2026-05-17T07:14:44Z)
• Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching [66.39914384073145]
  We propose a self-consistency framework that turns cheap diffusion-sampled reasoning into a reusable pool of step-level candidates.<n>We find that step-level recombination is most beneficial on harder problems.<n>Our training-free framework improves average accuracy by up to 2 across six math and coding tasks.
  arXiv  Detail & Related papers  (2026-02-26T11:08:39Z)
• Diagnosing LLM Reranker Behavior Under Fixed Evidence Pools [0.17188280334580197]
  Reranking evaluations study how a reranker orders candidates returned by an upstream retriever.<n>This setup couples ranking behavior with retrieval quality, so differences in output cannot be attributed to the ranking policy alone.<n>We introduce a controlled diagnostic that isolates reranking by using Multi-News clusters as fixed evidence pools.
  arXiv  Detail & Related papers  (2026-02-20T21:07:32Z)
• DiffuRank: Effective Document Reranking with Diffusion Language Models [71.16830004674513]
  We propose DiffuRank, a reranking framework built upon diffusion language models (dLLMs)<n>dLLMs support more flexible decoding and generation processes that are not constrained to a left-to-right order.<n>We show dLLMs achieve performance comparable to, and in some cases exceeding, that of autoregressive LLMs with similar model sizes.
  arXiv  Detail & Related papers  (2026-02-13T02:18:14Z)
• Dependence-Aware Label Aggregation for LLM-as-a-Judge via Ising Models [55.94503936470247]
  Large-scale AI evaluation increasingly relies on aggregating binary judgments from $K$ annotators, including judges.<n>Most classical methods assume annotators are conditionally independent given the true label $Yin0,1$, an assumption often violated by LLM judges.<n>We study label aggregation through a hierarchy of dependence-aware models based on Ising graphical models and latent factors.
  arXiv  Detail & Related papers  (2026-01-29T21:26:50Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• CompassVerifier: A Unified and Robust Verifier for LLMs Evaluation and Outcome Reward [50.97588334916863]
  We develop CompassVerifier, an accurate and robust lightweight verifier model for evaluation and outcome reward.<n>It demonstrates multi-domain competency spanning math, knowledge, and diverse reasoning tasks, with the capability to process various answer types.<n>We introduce VerifierBench benchmark comprising model outputs collected from multiple data sources, augmented through manual analysis of metaerror patterns to enhance CompassVerifier.
  arXiv  Detail & Related papers  (2025-08-05T17:55:24Z)
• GRP: Goal-Reversed Prompting for Zero-Shot Evaluation with LLMs [14.906150451947443]
  Using Large Language Models (LLMs) to evaluate and compare two answers typically involves having LLM-based judges select the better answer.<n>We propose a Goal-Reversed Prompting (GRP) approach for pairwise evaluation that shifts the original task from selecting the better answer to choosing the worse one.
  arXiv  Detail & Related papers  (2025-03-08T09:44:24Z)
• Inference Scaling for Bridging Retrieval and Augmented Generation [47.091086803980765]
  Retrieval-augmented generation (RAG) has emerged as a popular approach to steering the output of a large language model (LLM)<n>We show such bias can be mitigated, from inference scaling, aggregating inference calls from the permuted order of retrieved contexts.<n>We showcase the effectiveness of MOI on diverse RAG tasks, improving ROUGE-L on MS MARCO and EM on HotpotQA benchmarks by 7 points.
  arXiv  Detail & Related papers  (2024-12-14T05:06:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.