Integrating Local and Global Entropy for Uncertainty Quantification in LLMs

• URL: http://arxiv.org/abs/2606.09875v1
• Date: Tue, 02 Jun 2026 20:57:14 GMT
• Title: Integrating Local and Global Entropy for Uncertainty Quantification in LLMs
• Abstract summary: We show that hidden-state geometric entropy (global uncertainty) and token-level entropy (local uncertainty) are statistically near-orthogonal.<n>We propose Global-Local Uncertainty (GLU), an unsupervised, single-pass score that fuses the two signals via a multiplicative gate.
• Score: 28.285388648427737
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models hallucinate confidently, making uncertainty quantification (UQ) essential for reliable deployment. Existing methods rely predominantly on token-level signals, leaving the geometric structure of intermediate hidden states underused. In this paper, we take the geometric complexity of hidden-state matrices as a measure of the global uncertainty of LLMs, while treating token-level uncertainty estimation as a local metric. We show that hidden-state geometric entropy (global uncertainty) and token-level entropy (local uncertainty) are statistically near-orthogonal, capturing distinct failure regimes for reliability prediction. In particular, global geometry recovers the confident-but-wrong failure mode that local signals systematically miss. Building on this, we propose Global-Local Uncertainty (GLU), an unsupervised, single-pass score that fuses the two signals via a multiplicative gate. Across three model families and six benchmarks, GLU matches or outperforms all unsupervised baselines while requiring only a single forward pass and remaining length-normalized and architecture-agnostic.

Related papers

• Attention-Path Fragility as an Uncertainty Signal in Large Language Models [3.5772286398380255]
  We instantiate this as ASMI, a training-free estimator that masks attention heads and measures the BALD mutual information among the resultingworks.<n>On grounded QA an out-of-fold test shows it adds error-predictive information beyond single-pass confidence and entropy.<n>A head-level analysis shows that what tracks this boundary is not the presence of head-level fragility but whether that fragility couples to errors.
  arXiv  Detail & Related papers  (2026-08-11T16:59:02Z)
• Heteroskedastic Signals in Budgeted LLM Verification: Structural Heterogeneity Limits Optimization Gains [4.560961083890857]
  Large language model (LLM) systems increasingly use uncertainty signals to allocate limited computation across verification, test-time scaling, tool execution, and other selective-compute decisions.<n>We show that uncertainty quality is heteroskedastic across cost strata, with some regions exhibiting near-random discriminability despite concentrating many errors.
  arXiv  Detail & Related papers  (2026-06-14T14:45:57Z)
• Genuine Multipartite Nonlocality for Arbitrary Input: Maximal Randomness Generation and Robust Self-Testing [0.0]
  We introduce a Bell inequality capable of identifying genuine multipartite nonlocality (GMNL) in an arbitrary m-partite scenario.<n>We show that our framework enables the extraction of maximal global DI randomness (m bits) at the optimal quantum violation.
  arXiv  Detail & Related papers  (2026-06-09T14:45:31Z)
• Uncertainty-Aware Gaussian Map for Vision-Language Navigation [63.97713877754199]
  Vision-Language Navigation (VLN) requires an agent to navigate 3D environments following natural language instructions.<n>In this work, we explicitly model three forms of perceptual uncertainty (i.e., geometric, semantic, and appearance uncertainty) and integrate them into the agent's observation space to enable informed decision-making.
  arXiv  Detail & Related papers  (2026-05-26T03:33:47Z)
• Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering [8.56875523245176]
  Uncertainty Quantification is widely regarded as the primary safeguard for deploying Large Language Models (LLMs) in high-stakes domains.<n>We argue that the field suffers from a category error: mainstream UQ methods for LLMs are just unsupervised clustering algorithms.
  arXiv  Detail & Related papers  (2026-05-19T00:47:02Z)
• Discovery of Hidden Miscalibration Regimes [52.452902154360565]
  Models may be systematically overconfident on some kinds of inputs and underconfident on others.<n>We define the corresponding miscalibration field and propose a diagnostic framework for estimating it.<n>Our approach learns a calibration-aware representation of the input space and estimates signed local miscalibration by kernel smoothing in the learned geometry.
  arXiv  Detail & Related papers  (2026-05-13T13:07:50Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Adaptive Dual Uncertainty Optimization: Boosting Monocular 3D Object Detection under Test-Time Shifts [80.32933059529135]
  Test-Time Adaptation (TTA) methods have emerged to adapt to target distributions during inference.<n>We propose Dual Uncertainty Optimization (DUO), the first TTA framework designed to jointly minimize both uncertainties for robust M3OD.<n>In parallel, we design a semantic-aware normal field constraint that preserves geometric coherence in regions with clear semantic cues.
  arXiv  Detail & Related papers  (2025-08-28T07:09:21Z)
• Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities [79.9629927171974]
  Uncertainty in Large Language Models (LLMs) is crucial for applications where safety and reliability are important. We propose Kernel Language Entropy (KLE), a novel method for uncertainty estimation in white- and black-box LLMs.
  arXiv  Detail & Related papers  (2024-05-30T12:42:05Z)
• Many-body entropies and entanglement from polynomially-many local measurements [0.26388783516590225]
  We show that efficient estimation strategies exist under the assumption that all the spatial correlation lengths are finite. We argue that our method could be practically useful to detect bipartite mixed-state entanglement for large numbers of qubits available in today's quantum platforms.
  arXiv  Detail & Related papers  (2023-11-14T12:13:15Z)
• Identifying quantum many-body integrability and chaos using eigenstates trace distances [0.05999777817331316]
  We introduce an alternative indicator for quantum many-body integrability and chaos. It is based on the statistics of eigenstates by means of nearest-neighbor subsystem trace distances. We show that this provides us with a faithful classification through extensive numerical simulations.
  arXiv  Detail & Related papers  (2023-01-30T19:00:03Z)
• Global Convolutional Neural Processes [52.85558458799716]
  We build a member GloBal Convolutional Neural Process(GBCoNP) that achieves the SOTA log-likelihood in latent NPFs. It designs a global uncertainty representation p(z) which is an aggregation on a discretized input space. The learnt prior is analyzed on a variety of scenarios, including 1D, 2D, and a newly proposed spatial-temporal COVID dataset.
  arXiv  Detail & Related papers  (2021-09-02T03:32:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.