Calibrated Language Models Must Hallucinate

• URL: http://arxiv.org/abs/2311.14648v3
• Date: Wed, 20 Mar 2024 02:21:20 GMT
• Title: Calibrated Language Models Must Hallucinate
• Authors: Adam Tauman Kalai, Santosh S. Vempala,
• Abstract summary: Recent language models generate false but plausible-sounding text with surprising frequency. This work shows that there is an inherent statistical lower-bound on the rate that pretrained language models hallucinate certain types of facts. For "arbitrary" facts whose veracity cannot be determined from the training data, we show that hallucinations must occur at a certain rate for language models.
• Score: 11.891340760198798
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent language models generate false but plausible-sounding text with surprising frequency. Such "hallucinations" are an obstacle to the usability of language-based AI systems and can harm people who rely upon their outputs. This work shows that there is an inherent statistical lower-bound on the rate that pretrained language models hallucinate certain types of facts, having nothing to do with the transformer LM architecture or data quality. For "arbitrary" facts whose veracity cannot be determined from the training data, we show that hallucinations must occur at a certain rate for language models that satisfy a statistical calibration condition appropriate for generative language models. Specifically, if the maximum probability of any fact is bounded, we show that the probability of generating a hallucination is close to the fraction of facts that occur exactly once in the training data (a "Good-Turing" estimate), even assuming ideal training data without errors. One conclusion is that models pretrained to be sufficiently good predictors (i.e., calibrated) may require post-training to mitigate hallucinations on the type of arbitrary facts that tend to appear once in the training set. However, our analysis also suggests that there is no statistical reason that pretraining will lead to hallucination on facts that tend to appear more than once in the training data (like references to publications such as articles and books, whose hallucinations have been particularly notable and problematic) or on systematic facts (like arithmetic calculations). Therefore, different architectures and learning algorithms may mitigate these latter types of hallucinations.

Related papers

• Training Language Models on the Knowledge Graph: Insights on Hallucinations and Their Detectability [83.0884072598828]
  Hallucinations come in many forms, and there is no universally accepted definition. We focus on studying only those hallucinations where a correct answer appears verbatim in the training set. We find that for a fixed dataset, larger and longer-trained LMs hallucinate less. While we see detector size improves performance on fixed LM's outputs, we find an inverse relationship between the scale of the LM and the detectability of its hallucinations.
  arXiv  Detail & Related papers  (2024-08-14T23:34:28Z)
• Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models [65.32990889402927]
  We coin this phenomenon as knowledge overshadowing'' We show that the hallucination rate grows with both the imbalance ratio and the length of dominant condition description. We propose to utilize overshadowing conditions as a signal to catch hallucination before it is produced.
  arXiv  Detail & Related papers  (2024-07-10T20:37:42Z)
• Mitigating Large Language Model Hallucination with Faithful Finetuning [46.33663932554782]
  Large language models (LLMs) have demonstrated remarkable performance on various natural language processing tasks. They are prone to generating fluent yet untruthful responses, known as "hallucinations"
  arXiv  Detail & Related papers  (2024-06-17T07:16:07Z)
• On Large Language Models' Hallucination with Regard to Known Facts [74.96789694959894]
  Large language models are successful in answering factoid questions but are also prone to hallucination. We investigate the phenomenon of LLMs possessing correct answer knowledge yet still hallucinating from the perspective of inference dynamics. Our study shed light on understanding the reasons for LLMs' hallucinations on their known facts, and more importantly, on accurately predicting when they are hallucinating.
  arXiv  Detail & Related papers  (2024-03-29T06:48:30Z)
• Hallucinations in Neural Automatic Speech Recognition: Identifying Errors and Hallucinatory Models [11.492702369437785]
  Hallucinations are semantically unrelated to the source utterance, yet still fluent and coherent. We show that commonly used metrics, such as word error rates, cannot differentiate between hallucinatory and non-hallucinatory models. We devise a framework for identifying hallucinations by analysing their semantic connection with the ground truth and their fluency.
  arXiv  Detail & Related papers  (2024-01-03T06:56:56Z)
• Reducing Hallucinations in Neural Machine Translation with Feature Attribution [54.46113444757899]
  We present a case study focusing on model understanding and regularisation to reduce hallucinations in NMT. We first use feature attribution methods to study the behaviour of an NMT model that produces hallucinations. We then leverage these methods to propose a novel loss function that substantially helps reduce hallucinations and does not require retraining the model from scratch.
  arXiv  Detail & Related papers  (2022-11-17T20:33:56Z)
• Mutual Information Alleviates Hallucinations in Abstractive Summarization [73.48162198041884]
  We find a simple criterion under which models are significantly more likely to assign more probability to hallucinated content during generation: high model uncertainty. This finding offers a potential explanation for hallucinations: models default to favoring text with high marginal probability, when uncertain about a continuation. We propose a decoding strategy that switches to optimizing for pointwise mutual information of the source and target token--rather than purely the probability of the target token--when the model exhibits uncertainty.
  arXiv  Detail & Related papers  (2022-10-24T13:30:54Z)
• Inspecting the Factuality of Hallucinated Entities in Abstractive Summarization [36.052622624166894]
  State-of-the-art abstractive summarization systems often generate emphhallucinations; i.e., content that is not directly inferable from the source text. We propose a novel detection approach that separates factual from non-factual hallucinations of entities.
  arXiv  Detail & Related papers  (2021-08-30T15:40:52Z)
• On Hallucination and Predictive Uncertainty in Conditional Language Generation [76.18783678114325]
  Higher predictive uncertainty corresponds to a higher chance of hallucination. Epistemic uncertainty is more indicative of hallucination than aleatoric or total uncertainties. It helps to achieve better results of trading performance in standard metric for less hallucination with the proposed beam search variant.
  arXiv  Detail & Related papers  (2021-03-28T00:32:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.