Measuring the Impact of Lexical Training Data Coverage on Hallucination Detection in Large Language Models

• URL: http://arxiv.org/abs/2511.17946v1
• Date: Sat, 22 Nov 2025 06:59:55 GMT
• Title: Measuring the Impact of Lexical Training Data Coverage on Hallucination Detection in Large Language Models
• Authors: Shuo Zhang, Fabrizio Gotti, Fengran Mo, Jian-Yun Nie,
• Abstract summary: Hallucination in large language models (LLMs) is a fundamental challenge, particularly in open-domain question answering.<n>Prior work attempts to detect hallucination with model-internal signals such as token-level entropy or generation consistency.<n>We investigate whether data coverage itself can serve as a detection signal.
• Score: 26.89705770151822
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hallucination in large language models (LLMs) is a fundamental challenge, particularly in open-domain question answering. Prior work attempts to detect hallucination with model-internal signals such as token-level entropy or generation consistency, while the connection between pretraining data exposure and hallucination is underexplored. Existing studies show that LLMs underperform on long-tail knowledge, i.e., the accuracy of the generated answer drops for the ground-truth entities that are rare in pretraining. However, examining whether data coverage itself can serve as a detection signal is overlooked. We propose a complementary question: Does lexical training-data coverage of the question and/or generated answer provide additional signal for hallucination detection? To investigate this, we construct scalable suffix arrays over RedPajama's 1.3-trillion-token pretraining corpus to retrieve $n$-gram statistics for both prompts and model generations. We evaluate their effectiveness for hallucination detection across three QA benchmarks. Our observations show that while occurrence-based features are weak predictors when used alone, they yield modest gains when combined with log-probabilities, particularly on datasets with higher intrinsic model uncertainty. These findings suggest that lexical coverage features provide a complementary signal for hallucination detection. All code and suffix-array infrastructure are provided at https://github.com/WWWonderer/ostd.

Related papers

• Grounding or Guessing? Visual Signals for Detecting Hallucinations in Sign Language Translation [13.03365340564181]
  Hallucination is a major flaw in vision-language models and is particularly critical in sign language translation.<n>We propose a token-level reliability measure that quantifies how much the decoder uses visual information.<n>Our results show that reliability predicts hallucination rates, generalizes across datasets and architectures, and decreases under visual degradations.
  arXiv  Detail & Related papers  (2025-10-21T09:13:46Z)
• HalLoc: Token-level Localization of Hallucinations for Vision Language Models [36.12465376767014]
  Hallucinations pose a significant challenge to the reliability of large vision-language models.<n>HalLoc is a dataset designed for efficient, probabilistic hallucination detection.
  arXiv  Detail & Related papers  (2025-06-12T01:50:35Z)
• A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputs [71.97006967209539]
  Large Language Models (LLMs) have the tendency to hallucinate, i.e., to sporadically generate false or fabricated information.<n>Uncertainty quantification (UQ) provides a framework for assessing the reliability of model outputs.<n>We pre-train a collection of UQ heads for popular LLM series, including Mistral, Llama, and Gemma 2.
  arXiv  Detail & Related papers  (2025-05-13T03:30:26Z)
• HalluLens: LLM Hallucination Benchmark [49.170128733508335]
  Large language models (LLMs) often generate responses that deviate from user input or training data, a phenomenon known as "hallucination"<n>This paper introduces a comprehensive hallucination benchmark, incorporating both new extrinsic and existing intrinsic evaluation tasks.
  arXiv  Detail & Related papers  (2025-04-24T13:40:27Z)
• Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations [82.42811602081692]
  This paper introduces a subsequence association framework to systematically trace and understand hallucinations.<n>Key insight is hallucinations that arise when dominant hallucinatory associations outweigh faithful ones.<n>We propose a tracing algorithm that identifies causal subsequences by analyzing hallucination probabilities across randomized input contexts.
  arXiv  Detail & Related papers  (2025-04-17T06:34:45Z)
• Lookback Lens: Detecting and Mitigating Contextual Hallucinations in Large Language Models Using Only Attention Maps [48.58310785625051]
  Large language models (LLMs) can hallucinate details and respond with unsubstantiated answers. This paper describes a simple approach for detecting such contextual hallucinations.
  arXiv  Detail & Related papers  (2024-07-09T17:44:34Z)
• HypoTermQA: Hypothetical Terms Dataset for Benchmarking Hallucination Tendency of LLMs [0.0]
  Hallucinations pose a significant challenge to the reliability and alignment of Large Language Models (LLMs) This paper introduces an automated scalable framework that combines benchmarking LLMs' hallucination tendencies with efficient hallucination detection. The framework is domain-agnostic, allowing the use of any language model for benchmark creation or evaluation in any domain.
  arXiv  Detail & Related papers  (2024-02-25T22:23:37Z)
• AutoHall: Automated Hallucination Dataset Generation for Large Language Models [56.92068213969036]
  This paper introduces a method for automatically constructing model-specific hallucination datasets based on existing fact-checking datasets called AutoHall. We also propose a zero-resource and black-box hallucination detection method based on self-contradiction.
  arXiv  Detail & Related papers  (2023-09-30T05:20:02Z)
• Detecting Hallucinated Content in Conditional Neural Sequence Generation [165.68948078624499]
  We propose a task to predict whether each token in the output sequence is hallucinated (not contained in the input) We also introduce a method for learning to detect hallucinations using pretrained language models fine tuned on synthetic data.
  arXiv  Detail & Related papers  (2020-11-05T00:18:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.