Hallucination Detection: A Probabilistic Framework Using Embeddings Distance Analysis

• URL: http://arxiv.org/abs/2502.08663v1
• Date: Mon, 10 Feb 2025 09:44:13 GMT
• Title: Hallucination Detection: A Probabilistic Framework Using Embeddings Distance Analysis
• Authors: Emanuele Ricco, Lorenzo Cima, Roberto Di Pietro,
• Abstract summary: We introduce a mathematically sound methodology to reason about hallucination, and leverage it to build a tool to detect hallucinations.<n>To the best of our knowledge, we are the first to show that hallucinated content has structural differences with respect to correct content.<n>We leverage these structural differences to develop a tool to detect hallucinated responses, achieving an accuracy of 66% for a specific configuration of system parameters.
• Score: 2.089191490381739
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hallucinations are one of the major issues affecting LLMs, hindering their wide adoption in production systems. While current research solutions for detecting hallucinations are mainly based on heuristics, in this paper we introduce a mathematically sound methodology to reason about hallucination, and leverage it to build a tool to detect hallucinations. To the best of our knowledge, we are the first to show that hallucinated content has structural differences with respect to correct content. To prove this result, we resort to the Minkowski distances in the embedding space. Our findings demonstrate statistically significant differences in the embedding distance distributions, that are also scale free -- they qualitatively hold regardless of the distance norm used and the number of keywords, questions, or responses. We leverage these structural differences to develop a tool to detect hallucinated responses, achieving an accuracy of 66\% for a specific configuration of system parameters -- comparable with the best results in the field. In conclusion, the suggested methodology is promising and novel, possibly paving the way for further research in the domain, also along the directions highlighted in our future work.

Related papers

• Towards Long Context Hallucination Detection [49.195854802543714]
  Large Language Models (LLMs) have demonstrated remarkable performance across various tasks. They are prone to contextual hallucination, generating information that is either unsubstantiated or contradictory to the given context. We propose a novel architecture that enables pre-trained encoder models, such as BERT, to process long contexts and effectively detect contextual hallucinations.
  arXiv  Detail & Related papers  (2025-04-28T03:47:05Z)
• Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associations [82.42811602081692]
  This paper introduces a subsequence association framework to systematically trace and understand hallucinations. Key insight is hallucinations that arise when dominant hallucinatory associations outweigh faithful ones. We propose a tracing algorithm that identifies causal subsequences by analyzing hallucination probabilities across randomized input contexts.
  arXiv  Detail & Related papers  (2025-04-17T06:34:45Z)
• Can Your Uncertainty Scores Detect Hallucinated Entity? [14.432545893757677]
  We propose a new data set, HalluEntity, which annotates hallucination at the entity level. Based on the dataset, we evaluate uncertainty-based hallucination detection approaches across 17 modern LLMs. Our experimental results show that uncertainty estimation approaches focusing on individual token probabilities tend to over-predict hallucinations.
  arXiv  Detail & Related papers  (2025-02-17T16:01:41Z)
• Who Brings the Frisbee: Probing Hidden Hallucination Factors in Large Vision-Language Model via Causality Analysis [14.033320167387194]
  A major challenge in their real-world application is hallucination, where LVLMs generate non-existent visual elements, eroding user trust.<n>We hypothesize that hidden factors, such as objects, contexts, and semantic foreground-background structures, induce hallucination.<n>By analyzing the causality between images, text prompts, and network saliency, we systematically explore interventions to block these factors.
  arXiv  Detail & Related papers  (2024-12-04T01:23:57Z)
• Hallucination Detection: Robustly Discerning Reliable Answers in Large Language Models [70.19081534515371]
  Large Language Models (LLMs) have gained widespread adoption in various natural language processing tasks. They generate unfaithful or inconsistent content that deviates from the input source, leading to severe consequences. We propose a robust discriminator named RelD to effectively detect hallucination in LLMs' generated answers.
  arXiv  Detail & Related papers  (2024-07-04T18:47:42Z)
• On Large Language Models' Hallucination with Regard to Known Facts [74.96789694959894]
  Large language models are successful in answering factoid questions but are also prone to hallucination. We investigate the phenomenon of LLMs possessing correct answer knowledge yet still hallucinating from the perspective of inference dynamics. Our study shed light on understanding the reasons for LLMs' hallucinations on their known facts, and more importantly, on accurately predicting when they are hallucinating.
  arXiv  Detail & Related papers  (2024-03-29T06:48:30Z)
• In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation [36.31646727970656]
  Large language models (LLMs) frequently hallucinate and produce factual errors. correct generations tend to have sharper context activations in the hidden states of the in-context tokens, compared to the incorrect ones. We propose an entropy-based metric to quantify the sharpness'' among the in-context hidden states and incorporate it into the decoding process.
  arXiv  Detail & Related papers  (2024-03-03T15:53:41Z)
• DelucionQA: Detecting Hallucinations in Domain-specific Question Answering [22.23664008053246]
  Hallucination is a well-known phenomenon in text generated by large language models (LLMs) We introduce a dataset, DelucionQA, that captures hallucinations made by retrieval-augmented LLMs for a domain-specific QA task. We propose a set of hallucination detection methods to serve as baselines for future works from the research community.
  arXiv  Detail & Related papers  (2023-12-08T17:41:06Z)
• HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data [102.56792377624927]
  hallucinations inherent in machine-generated data remain under-explored. We present a novel hallucination detection and elimination framework, HalluciDoctor, based on the cross-checking paradigm. Our method successfully mitigates 44.6% hallucinations relatively and maintains competitive performance compared to LLaVA.
  arXiv  Detail & Related papers  (2023-11-22T04:52:58Z)
• On Hallucination and Predictive Uncertainty in Conditional Language Generation [76.18783678114325]
  Higher predictive uncertainty corresponds to a higher chance of hallucination. Epistemic uncertainty is more indicative of hallucination than aleatoric or total uncertainties. It helps to achieve better results of trading performance in standard metric for less hallucination with the proposed beam search variant.
  arXiv  Detail & Related papers  (2021-03-28T00:32:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.