Pre-Generation Hallucination Detection in Large Language Models via Soft-Target Attention Probing

• URL: http://arxiv.org/abs/2606.21917v1
• Date: Sat, 20 Jun 2026 07:31:41 GMT
• Title: Pre-Generation Hallucination Detection in Large Language Models via Soft-Target Attention Probing
• Abstract summary: We introduce soft-target supervision based on the empirical answer error rate overally sampled outputs.<n>We adapt attention probing to the pre-generation setting, enabling the detector to selectively aggregate hallucination-relevant prompt representations.
• Score: 0.9870126088784975
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting hallucination risk before generation enables abstention, retrieval augmentation, and routing decisions without incurring the cost of decoding. While prior work has shown that such risk can be estimated from a model's internal representations, existing approaches treat this as binary classification over a single decoded output. We instead formulate it as a risk-estimation problem. Under this formulation, we introduce soft-target supervision based on the empirical answer error rate over stochastically sampled outputs - an estimator we prove to be the unique unbiased minimum-variance estimator of the model's per-prompt error probability under its sampling distribution. We further adapt attention probing to the pre-generation setting, enabling the detector to selectively aggregate hallucination-relevant prompt representations. Across three question-answering benchmarks and five models, attention probing outperforms linear probing on short-answer tasks. Replacing binary labels with soft-target supervision further and consistently improves detection quality.

Related papers

• Open-Set Visual Text Forensics via Sparse-Constraint Rectified Flow [11.434432702589868]
  Generative AI enables sophisticated visual text manipulations that increasingly evade current forensic detectors.<n>We propose a generative detector that localizes tampering by estimating the local restoration cost required to align a query image with authentic visual-text statistics.<n>Our method achieves state-of-the-art performance, surpassing the runner-up by 3.2 and 4.8 percentage points in F1 and IoU, respectively.
  arXiv  Detail & Related papers  (2026-08-03T14:03:27Z)
• Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Models [53.15468578562038]
  We propose an Uncertainty-aware Exploratory Direct Preference Optimization (UE-DPO) method for Multimodal Large Language Models (MLLMs)<n>We first quantify the uncertainty from the model's failure to ground token predictions in the given image.<n>We then encourage more learning pressure on visually deficient tokens in preferred samples, and alleviate the over-penalization of beneficial knowledge in dispreferred samples.
  arXiv  Detail & Related papers  (2026-05-06T13:08:12Z)
• Entropy Alone is Insufficient for Safe Selective Prediction in LLMs [20.664633053172327]
  Selective prediction systems can mitigate harms resulting from language model hallucinations by abstaining from answering in high-risk cases.<n>Uncertainty quantification techniques are often employed to identify such cases, but are rarely evaluated in the context of the wider selective prediction policy.<n>We identify a model-dependent failure mode of entropy-based uncertainty methods that leads to unreliable abstention behaviour, and address it by combining entropy scores with a correctness probe signal.
  arXiv  Detail & Related papers  (2026-03-22T11:27:13Z)
• From Calibration to Refinement: Seeking Certainty via Probabilistic Evidence Propagation for Noisy-Label Person Re-Identification [40.73759251488672]
  Existing noise-robust person Re-ID methods rely on loss-correction or sample-selection strategies using softmax outputs.<n>We propose the CAlibration-to-REfinement (CARE) method, a two-stage framework that seeks certainty through probabilistic evidence propagation from calibration to refinement.<n>In the refinement stage, we design the evidence propagation refinement (EPR) that can more accurately distinguish between clean and noisy samples.
  arXiv  Detail & Related papers  (2026-02-26T15:50:15Z)
• Catching Contamination Before Generation: Spectral Kill Switches for Agents [0.0]
  We introduce a diagnostic that uses only the forward pass to emit a binary accept or reject signal during agent execution.<n>The method analyzes token graphs induced by attention and computes two spectral statistics in early layers.<n>We show that a single threshold on the high frequency energy ratio is optimal in the Bayes sense for detecting context inconsistency.
  arXiv  Detail & Related papers  (2025-11-08T02:24:05Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.<n>We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.<n>Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Token-Level Adversarial Prompt Detection Based on Perplexity Measures and Contextual Information [67.78183175605761]
  Large Language Models are susceptible to adversarial prompt attacks. This vulnerability underscores a significant concern regarding the robustness and reliability of LLMs. We introduce a novel approach to detecting adversarial prompts at a token level.
  arXiv  Detail & Related papers  (2023-11-20T03:17:21Z)
• Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations [67.40641255908443]
  We identify limitations of model-randomization-based sanity checks for the purpose of evaluating explanations. Top-down model randomization preserves scales of forward pass activations with high probability.
  arXiv  Detail & Related papers  (2022-11-22T18:52:38Z)
• Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
  Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates. We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters. We show that our method improves both accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2021-09-27T01:09:08Z)
• Self-Paced Uncertainty Estimation for One-shot Person Re-Identification [9.17071384578203]
  We propose a novel Self-Paced Uncertainty Estimation Network (SPUE-Net) for one-shot Person Re-ID. By introducing a self-paced sampling strategy, our method can estimate the pseudo-labels of unlabeled samples iteratively to expand the labeled samples. In addition, we apply a Co-operative learning method of local uncertainty estimation combined with determinacy estimation to achieve better hidden space feature mining.
  arXiv  Detail & Related papers  (2021-04-19T09:20:30Z)
• Understanding Classifier Mistakes with Generative Models [88.20470690631372]
  Deep neural networks are effective on supervised learning tasks, but have been shown to be brittle. In this paper, we leverage generative models to identify and characterize instances where classifiers fail to generalize. Our approach is agnostic to class labels from the training set which makes it applicable to models trained in a semi-supervised way.
  arXiv  Detail & Related papers  (2020-10-05T22:13:21Z)
• Pairwise Discriminative Neural PLDA for Speaker Verification [41.76303371621405]
  We propose a Pairwise neural discriminative model for the task of speaker verification. We construct a differentiable cost function which approximates speaker verification loss. Experiments are performed on the NIST SRE 2018 development and evaluation datasets.
  arXiv  Detail & Related papers  (2020-01-20T09:52:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.