When Do Supervised UQ Ensembles Improve LLM Hallucination Detection? A Robustness Study

• URL: http://arxiv.org/abs/2608.24492v1
• Date: Tue, 25 Aug 2026 12:43:07 GMT
• Title: When Do Supervised UQ Ensembles Improve LLM Hallucination Detection? A Robustness Study
• Abstract summary: Uncertainty quantification (UQ) methods are widely used for hallucination detection in large language models (LLMs)<n>We study a supervised ensembling framework that trains a classifier over heterogeneous UQ-based scorer outputs.<n>We find that supervised ensembles outperform the best individual scorer in 30 of 32 settings, with gains realized from as few as 100 labeled instances.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Uncertainty quantification (UQ) methods are widely used for hallucination detection in large language models (LLMs) in closed-book settings where ground-truth evidence is unavailable at inference time. Prior work has proposed combining UQ signals via learned ensembles, but empirical investigations into the robustness of these ensembles are limited. We study a supervised ensembling framework that trains a classifier over heterogeneous UQ-based scorer outputs on a small, domain-specific dataset of labeled LLM responses, then applies it to out-of-sample hallucination classification without retrieval, tools, or reference documents. Across four LLMs, nine datasets, and three generation regimes (short-form QA, long-form generation, and code generation), we provide a systematic robustness analysis along three axes: sample efficiency, in-domain dataset transfer, and generation regime dependence. We find that supervised ensembles outperform the best individual scorer in 30 of 32 settings, with gains realized from as few as 100 labeled instances. Ensembles retain most of their advantage in cases of in-domain transfer under distribution shift, outperforming the best non-ensemble scorer in 23 of 28 transfer settings. Sampling-based black-box ensembles are nearly as effective as full ensembles, while single-generation white-box ensembles offer limited benefit.

Related papers

• Unsupervised Transfer Clustering for Mitigating Cold Start in Active Prompt Learning [1.5539424546888112]
  Vision-Language Models (VLMs) are able to achieve impressive zero-shot classification performance by aligning visual and textual representations.<n>Each new task still demands handcrafted prompts.<n>Active Prompt Learning (APL) combines Active Learning (AL) and Prompt Learning (PL) into a single framework.
  arXiv  Detail & Related papers  (2026-09-04T18:14:44Z)
• LLM-based Schema-Guided Extraction and Validation of Missing-Person Intelligence from Heterogeneous Data Sources [0.7734726150561088]
  Missing-person and child-safety investigations rely on heterogeneous case documents, including structured forms, bulletin-style posters, and narrative web profiles.<n>Variations in layout, terminology, and data quality impede rapid triage, large-scale analysis, and search-planning.<n>This paper introduces the Guardian Pack, an AI-driven parsing and normalization pipeline that transforms multi-source investigative documents into a unified, schema-compliant representation.
  arXiv  Detail & Related papers  (2026-04-08T01:35:56Z)
• Benchmarking Few-shot Transferability of Pre-trained Models with Improved Evaluation Protocols [123.73663884421272]
  Few-shot transfer has been revolutionized by stronger pre-trained models and improved adaptation algorithms.<n>We establish FEWTRANS, a comprehensive benchmark containing 10 diverse datasets.<n>By releasing FEWTRANS, we aim to provide a rigorous "ruler" to streamline reproducible advances in few-shot transfer learning research.
  arXiv  Detail & Related papers  (2026-02-28T05:41:57Z)
• Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Study [0.0]
  We introduce a taxonomy for fine-grained uncertainty quantification in long-form LLM outputs.<n>We formalize several families of consistency-based black-box scorers, providing generalizations and extensions of existing methods.<n>We find 1) claim-response entailment consistently performs better or on par with more complex claim-level scorers, 2) claim-level scoring generally yields better results than sentence-level scoring, and 3) uncertainty-aware decoding is highly effective for improving the factuality of long-form outputs.
  arXiv  Detail & Related papers  (2026-02-19T15:02:29Z)
• Test-time Verification via Optimal Transport: Coverage, ROC, & Sub-optimality [53.03186946689658]
  Test-time scaling with verification has shown promise in improving the performance of large language models.<n>The effect of verification manifests through interactions of three quantities: (i) the generator's coverage, (ii) the verifier's region of convergence (ROC), and (iii) the sampling algorithm's sub-optimality.<n>We frame verifiable test-time scaling as a transport problem. This characterizes the interaction of coverage, ROC, and sub-optimality.
  arXiv  Detail & Related papers  (2025-10-21T18:05:42Z)
• Learned Hallucination Detection in Black-Box LLMs using Token-level Entropy Production Rate [0.19676943624884313]
  Hallucinations in Large Language Model (LLM) outputs for Question Answering tasks critically undermine their real-world reliability.<n>This paper introduces an applied methodology for robust, one-shot hallucination detection, specifically designed for scenarios with limited data access.<n>Our approach derives uncertainty indicators directly from these readily available log-probabilities generated during non-greedy decoding.
  arXiv  Detail & Related papers  (2025-09-01T13:34:21Z)
• Uncertainty Quantification for Language Models: A Suite of Black-Box, White-Box, LLM Judge, and Ensemble Scorers [0.0]
  Hallucinations are a persistent problem with Large Language Models (LLMs)<n>We outline a versatile framework for zero-resource hallucination detection that practitioners can apply to real-world use cases.<n>To enhance flexibility, we propose a tunable ensemble approach that incorporates any combination of the individual confidence scores.
  arXiv  Detail & Related papers  (2025-04-27T14:24:45Z)
• Project-Probe-Aggregate: Efficient Fine-Tuning for Group Robustness [61.45587642780908]
  We propose a three-step approach for parameter-efficient fine-tuning of image-text foundation models.<n>Our method improves its two key components: minority samples identification and the robust training algorithm.<n>Our theoretical analysis shows that our PPA enhances minority group identification and is Bayes optimal for minimizing the balanced group error.
  arXiv  Detail & Related papers  (2025-03-12T15:46:12Z)
• Add-One-In: Incremental Sample Selection for Large Language Models via a Choice-Based Greedy Paradigm [50.492124556982674]
  This paper introduces a novel choice-based sample selection framework.<n>It shifts the focus from evaluating individual sample quality to comparing the contribution value of different samples.<n>We validate our approach on a larger medical dataset, highlighting its practical applicability in real-world applications.
  arXiv  Detail & Related papers  (2025-03-04T07:32:41Z)
• A Systematic Examination of Preference Learning through the Lens of Instruction-Following [83.71180850955679]
  We use a novel synthetic data generation pipeline to generate 48,000 instruction unique-following prompts.<n>With our synthetic prompts, we use two preference dataset curation methods - rejection sampling (RS) and Monte Carlo Tree Search (MCTS)<n>Experiments reveal that shared prefixes in preference pairs, as generated by MCTS, provide marginal but consistent improvements.<n>High-contrast preference pairs generally outperform low-contrast pairs; however, combining both often yields the best performance.
  arXiv  Detail & Related papers  (2024-12-18T15:38:39Z)
• A Framework for Fine-Tuning LLMs using Heterogeneous Feedback [69.51729152929413]
  We present a framework for fine-tuning large language models (LLMs) using heterogeneous feedback. First, we combine the heterogeneous feedback data into a single supervision format, compatible with methods like SFT and RLHF. Next, given this unified feedback dataset, we extract a high-quality and diverse subset to obtain performance increases.
  arXiv  Detail & Related papers  (2024-08-05T23:20:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.