Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification

• URL: http://arxiv.org/abs/2604.17112v1
• Date: Sat, 18 Apr 2026 19:00:27 GMT
• Title: Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification
• Abstract summary: Large language models (LLMs) often produce confident yet incorrect responses.<n>Self-consistency to estimate aleatoric uncertainty (AU) collapses when models are overconfident.<n>We show that cross-model semantic disagreement is higher on incorrect answers precisely when AU is low.
• Score: 41.04503562937318
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) often produce confident yet incorrect responses, and uncertainty quantification is one potential solution to more robust usage. Recent works routinely rely on self-consistency to estimate aleatoric uncertainty (AU), yet this proxy collapses when models are overconfident and produce the same incorrect answer across samples. We analyze this regime and show that cross-model semantic disagreement is higher on incorrect answers precisely when AU is low. Motivated by this, we introduce an epistemic uncertainty (EU) term that operates in the black-box access setting: EU uses only generated text from a small, scale-matched ensemble and is computed as the gap between inter-model and intra-model sequence-semantic similarity. We then define total uncertainty (TU) as the sum of AU and EU. In a comprehensive study across five 7-9B instruction-tuned models and ten long-form tasks, TU improves ranking calibration and selective abstention relative to AU, and EU reliably flags confident failures where AU is low. We further characterize when EU is most useful via agreement and complementarity diagnostics.

Related papers

• Inference-Time Steering for Cross-Lingual Factual Consistency in LLMs [51.56484100374058]
  Large Language Models (LLMs) demonstrate remarkable multilingual fluency, but their internal knowledge representations remain disproportionately biased toward high-resource languages.<n>We investigate whether these biases can be mitigated at inference time, forcing an English-prompted model to answer as if it were queried in target languages.<n>Experiments on Gemma 3 12B Instruct reveal persona prompting to be the strongest overall intervention, balancing efficacy, safety, and out-of-domain generalization.
  arXiv  Detail & Related papers  (2026-07-21T16:15:05Z)
• Uncertainty Is Not a Safety Net for Clinical VQA, but Can It Anticipate Model Failure? [33.91781729231922]
  We benchmark 8 methods across 12 vision-language models (VLMs) on clinical visual question-answering (VQA)<n>We find that UE quality is not an intrinsic property of the UE method: it tracks model accuracy, degrading precisely where the model performance is weakest.<n>Our results position UE as a diagnostic tool for identifying fragile predictions and motivate perturbation-based evaluation as a path toward safe clinical deployment.
  arXiv  Detail & Related papers  (2026-06-15T11:27:11Z)
• UAU-Net: Uncertainty-aware Representation Learning and Evidential Classification for Facial Action Unit Detection [9.627381138509422]
  UAU-Net is an Uncertainty-aware AU detection framework that explicitly models uncertainty at both stages.<n>Experiments on BP4D and DISFA show that UAU-Net achieves strong AU detection performance.
  arXiv  Detail & Related papers  (2026-04-23T02:47:24Z)
• Sensitivity Uncertainty Alignment in Large Language Models [0.0]
  We propose a framework for analyzing failures of large language models under adversarial and ambiguous inputs.<n>We define a scalar score, SUA_theta(x), capturing the difference between distributional sensitivity and predictive entropy.<n>We introduce SUA-TR, a training method combining consistency regularization and entropy alignment, along with an abstention rule for safer inference.
  arXiv  Detail & Related papers  (2026-04-21T17:53:12Z)
• Towards Reliable Truth-Aligned Uncertainty Estimation in Large Language Models [54.51264434040939]
  Uncertainty estimation (UE) aims to detect hallucinated outputs of large language models (LLMs)<n>We formalise this phenomenon as proxy failure, since most UE metrics originate from model behaviour.<n>We propose Truth AnChoring (TAC), a post-hoc calibration method to remedy UE metrics.
  arXiv  Detail & Related papers  (2026-04-01T03:42:16Z)
• Rethinking Uncertainty Quantification and Entanglement in Image Segmentation [8.952491535016444]
  Uncertainty (UQ) is crucial in safety-critical applications such as medical image segmentation.<n>Many methods exist for modeling AU (such as Probabilistic UNet, Diffusion) and EU (such as ensembles, MC Dropout) but it is unclear how they interact when combined.<n>We present a comprehensive empirical study covering a broad range of AU-EU model combinations, propose a metric to quantify uncertainty entanglement, and evaluate both across downstream UQ tasks.
  arXiv  Detail & Related papers  (2026-03-19T11:43:26Z)
• Decomposed Prompting Does Not Fix Knowledge Gaps, But Helps Models Say "I Don't Know" [47.930782177987446]
  Large language models often struggle to recognize their knowledge limits in closed-book question answering, leading to confident hallucinations.<n>We evaluate three task-equivalent prompting regimes: Direct, Assistive, and Incremental, across different model scales and multi-hop QA benchmarks.<n>Because factual knowledge is stable while hallucinations are agreement, cross-regime provides a precise signal of internal uncertainty.
  arXiv  Detail & Related papers  (2026-02-04T18:39:58Z)
• The Illusion of Certainty: Uncertainty quantification for LLMs fails under ambiguity [48.899855816199484]
  We introduce MAQA* and AmbigQA*, the first ambiguous question-answering (QA) datasets equipped with ground-truth answer distributions.<n>We show that predictive-distribution and ensemble-based estimators are fundamentally limited under ambiguity.
  arXiv  Detail & Related papers  (2025-11-06T14:46:35Z)
• Why Uncertainty Estimation Methods Fall Short in RAG: An Axiomatic Analysis [15.553942864736989]
  Uncertainty Estimation (UE) quantifies the model's confidence and helps users assess response reliability.<n>This paper shows that current UE methods cannot reliably assess correctness in the Retrieval-Augmented Generation setting.<n>We propose an axiomatic framework to identify deficiencies in existing methods and guide the development of improved approaches.
  arXiv  Detail & Related papers  (2025-05-12T11:47:42Z)
• Uncertainty Estimation of Large Language Models in Medical Question Answering [60.72223137560633]
  Large Language Models (LLMs) show promise for natural language generation in healthcare, but risk hallucinating factually incorrect information. We benchmark popular uncertainty estimation (UE) methods with different model sizes on medical question-answering datasets. Our results show that current approaches generally perform poorly in this domain, highlighting the challenge of UE for medical applications.
  arXiv  Detail & Related papers  (2024-07-11T16:51:33Z)
• Detecting Multimodal Situations with Insufficient Context and Abstaining from Baseless Predictions [75.45274978665684]
  Vision-Language Understanding (VLU) benchmarks contain samples where answers rely on assumptions unsupported by the provided context.<n>We collect contextual data for each sample whenever available and train a context selection module to facilitate evidence-based model predictions.<n>We develop a general-purpose Context-AwaRe Abstention detector to identify samples lacking sufficient context and enhance model accuracy.
  arXiv  Detail & Related papers  (2024-05-18T02:21:32Z)
• Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguation [5.55197751179213]
  Existing supervised methods treat WSD as a classification task and have achieved remarkable performance. This paper extensively studies uncertainty estimation (UE) on the benchmark designed for WSD. We examine the capability of capturing data and model uncertainties by the model with the selected UE score on well-designed test scenarios and discover that the model reflects data uncertainty satisfactorily but underestimates model uncertainty.
  arXiv  Detail & Related papers  (2023-05-22T15:18:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.