SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation

• URL: http://arxiv.org/abs/2607.05721v1
• Date: Tue, 07 Jul 2026 01:09:46 GMT
• Title: SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generation
• Abstract summary: We introduce SPANUQ, a lightweight probe that distills the uncertainty knowledge from expensive multi-sample inference into a single forward pass over hidden states.<n>We construct SPANUQ-BENCH, the first span-level uncertainty benchmark comprising 20K prompts, 293K annotated spans, and continuous soft labels.<n>Experiments show that SPANUQ consistently achieves the best span-level uncertainty quality, outperforming the strongest probe baseline and all sampling-based methods.
• Score: 39.25766957943906
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Uncertainty estimation is essential not only for the trustworthy deployment of large language models (LLMs) but also as a foundation for self-refinement in LLM generation. However, existing approaches operate at suboptimal granularities: token-level scores lack semantic coherence, while sequence-level scores fail to localize errors. We formalize Span-Level Uncertainty Estimation (SLUE), a new task that targets the natural granularity for uncertainty: semantically coherent text spans, each conveying a single assessable unit of meaning. To address this task, we introduce SPANUQ, a lightweight probe that distills the uncertainty knowledge from expensive multi-sample inference into a single forward pass over LLM hidden states. SPANUQ employs a DETR-style span decoder to simultaneously detect spans and estimate their uncertainty via a Mixture of Beta distribution, trained with a principled combination of Beta NLL regression and contrastive ranking objectives. We construct SPANUQ-BENCH, the first span-level uncertainty benchmark comprising 20K prompts, 293K annotated spans, and continuous soft labels derived from multi-sample claim verification. Experiments on five LLM backbones show that SPANUQ consistently achieves the best span-level uncertainty quality, outperforming the strongest probe baseline and all sampling-based methods while being 10-20x faster. Its DETR-based span detector attains 0.910 F1, surpassing the best heuristic by 39.4%, enabling precise error localization that sequence-level methods cannot provide. The framework generalizes across five LLMs spanning two model families.

Related papers

• How Confident Is the First Token? An Uncertainty-Calibrated Prompt Optimization Framework for Large Language Model Classification and Understanding [4.852493246050006]
  Large language models (LLMs) generate outputs autoregressively, leading to inevitable output uncertainty.<n>We propose Log-Scale Focal Uncertainty (LSFU), a first-token-based metric inspired by focal loss.<n>We show UCPOF improves average accuracy by 6.03% over few-shot baselines, surpasses always-on full RAG by 5.75%, and reduces the average retrieval trigger rate by 50.66%.
  arXiv  Detail & Related papers  (2026-02-23T11:55:14Z)
• Can Large Language Models Express Uncertainty Like Human? [71.27418419522884]
  We release the first diverse, large-scale dataset of hedging expressions with human-annotated confidence scores.<n>We conduct the first systematic study of linguistic confidence across modern large language models.
  arXiv  Detail & Related papers  (2025-09-29T02:34:30Z)
• On Verbalized Confidence Scores for LLMs [25.160810008907397]
  Uncertainty quantification for large language models (LLMs) can establish more human trust into their responses.<n>This work focuses on asking the LLM itself to verbalize its uncertainty with a confidence score as part of its output tokens.<n>We assess the reliability of verbalized confidence scores with respect to different datasets, models, and prompt methods.
  arXiv  Detail & Related papers  (2024-12-19T11:10:36Z)
• CLUE: Concept-Level Uncertainty Estimation for Large Language Models [49.92690111618016]
  We propose a novel framework for Concept-Level Uncertainty Estimation for Large Language Models (LLMs) We leverage LLMs to convert output sequences into concept-level representations, breaking down sequences into individual concepts and measuring the uncertainty of each concept separately. We conduct experiments to demonstrate that CLUE can provide more interpretable uncertainty estimation results compared with sentence-level uncertainty.
  arXiv  Detail & Related papers  (2024-09-04T18:27:12Z)
• Unconditional Truthfulness: Learning Unconditional Uncertainty of Large Language Models [104.55763564037831]
  We train a regression model that leverages attention maps, probabilities on the current generation step, and recurrently computed uncertainty scores from previously generated tokens.<n>Our evaluation shows that the proposed method is highly effective for selective generation, achieving substantial improvements over rivaling unsupervised and supervised approaches.
  arXiv  Detail & Related papers  (2024-08-20T09:42:26Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Self-Evaluation Improves Selective Generation in Large Language Models [54.003992911447696]
  We reformulate open-ended generation tasks into token-level prediction tasks. We instruct an LLM to self-evaluate its answers. We benchmark a range of scoring methods based on self-evaluation.
  arXiv  Detail & Related papers  (2023-12-14T19:09:22Z)
• Boosting Cross-Lingual Transfer via Self-Learning with Uncertainty Estimation [34.97086123805344]
  Recent multilingual pre-trained language models have achieved remarkable zero-shot performance. We propose a self-learning framework that further utilizes unlabeled data of target languages. We evaluate our framework with uncertainties on two cross-lingual tasks including Named Entity Recognition (NER) and Natural Language Inference (NLI) covering 40 languages in total.
  arXiv  Detail & Related papers  (2021-09-01T05:26:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.