Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models

• URL: http://arxiv.org/abs/2407.08039v1
• Date: Wed, 10 Jul 2024 20:37:42 GMT
• Title: Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models
• Authors: Yuji Zhang, Sha Li, Jiateng Liu, Pengfei Yu, Yi R. Fung, Jing Li, Manling Li, Heng Ji,
• Abstract summary: We coin this phenomenon as knowledge overshadowing'' We show that the hallucination rate grows with both the imbalance ratio and the length of dominant condition description. We propose to utilize overshadowing conditions as a signal to catch hallucination before it is produced.
• Score: 65.32990889402927
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hallucination is often regarded as a major impediment for using large language models (LLMs), especially for knowledge-intensive tasks. Even when the training corpus consists solely of true statements, language models still generate hallucinations in the form of amalgamations of multiple facts. We coin this phenomenon as ``knowledge overshadowing'': when we query knowledge from a language model with multiple conditions, some conditions overshadow others, leading to hallucinated outputs. This phenomenon partially stems from training data imbalance, which we verify on both pretrained models and fine-tuned models, over a wide range of LM model families and sizes.From a theoretical point of view, knowledge overshadowing can be interpreted as over-generalization of the dominant conditions (patterns). We show that the hallucination rate grows with both the imbalance ratio (between the popular and unpopular condition) and the length of dominant condition description, consistent with our derived generalization bound. Finally, we propose to utilize overshadowing conditions as a signal to catch hallucination before it is produced, along with a training-free self-contrastive decoding method to alleviate hallucination during inference. Our proposed approach showcases up to 82% F1 for hallucination anticipation and 11.2% to 39.4% hallucination control, with different models and datasets.

Related papers

• Distinguishing Ignorance from Error in LLM Hallucinations [43.62904897907926]
  We focus on close-book Question Answering (CBQA), where previous work has not fully addressed the distinction between two possible kinds of hallucinations. We argue that distinguishing these cases is crucial for detecting and mitigating hallucinations.
  arXiv  Detail & Related papers  (2024-10-29T14:31:33Z)
• VideoHallucer: Evaluating Intrinsic and Extrinsic Hallucinations in Large Video-Language Models [59.05674402770661]
  This work introduces VideoHallucer, the first comprehensive benchmark for hallucination detection in large video-language models (LVLMs) VideoHallucer categorizes hallucinations into two main types: intrinsic and extrinsic, offering further subcategories for detailed analysis.
  arXiv  Detail & Related papers  (2024-06-24T06:21:59Z)
• On Large Language Models' Hallucination with Regard to Known Facts [74.96789694959894]
  Large language models are successful in answering factoid questions but are also prone to hallucination. We investigate the phenomenon of LLMs possessing correct answer knowledge yet still hallucinating from the perspective of inference dynamics. Our study shed light on understanding the reasons for LLMs' hallucinations on their known facts, and more importantly, on accurately predicting when they are hallucinating.
  arXiv  Detail & Related papers  (2024-03-29T06:48:30Z)
• Unfamiliar Finetuning Examples Control How Language Models Hallucinate [75.03210107477157]
  Large language models are known to hallucinate when faced with unfamiliar queries. We find that unfamiliar examples in the models' finetuning data are crucial in shaping these errors. Our work further investigates RL finetuning strategies for improving the factuality of long-form model generations.
  arXiv  Detail & Related papers  (2024-03-08T18:28:13Z)
• Quantity Matters: Towards Assessing and Mitigating Number Hallucination in Large Vision-Language Models [57.42800112251644]
  We focus on a specific type of hallucination-number hallucination, referring to models incorrectly identifying the number of certain objects in pictures. We devise a training approach aimed at improving consistency to reduce number hallucinations, which leads to an 8% enhancement in performance over direct finetuning methods.
  arXiv  Detail & Related papers  (2024-03-03T02:31:11Z)
• Calibrated Language Models Must Hallucinate [11.891340760198798]
  Recent language models generate false but plausible-sounding text with surprising frequency. This work shows that there is an inherent statistical lower-bound on the rate that pretrained language models hallucinate certain types of facts. For "arbitrary" facts whose veracity cannot be determined from the training data, we show that hallucinations must occur at a certain rate for language models.
  arXiv  Detail & Related papers  (2023-11-24T18:29:50Z)
• Evaluating Hallucinations in Chinese Large Language Models [65.4771562909392]
  We establish a benchmark named HalluQA (Chinese Hallucination Question-Answering) to measure the hallucination phenomenon in Chinese large language models. We consider two types of hallucinations: imitative falsehoods and factual errors, and we construct adversarial samples based on GLM-130B and ChatGPT. For evaluation, we design an automated evaluation method using GPT-4 to judge whether a model output is hallucinated.
  arXiv  Detail & Related papers  (2023-10-05T07:57:09Z)
• Understanding and Detecting Hallucinations in Neural Machine Translation via Model Introspection [28.445196622710164]
  We first identify internal model symptoms of hallucinations by analyzing the relative token contributions to the generation in contrastive hallucinated vs. non-hallucinated outputs generated via source perturbations. We then show that these symptoms are reliable indicators of natural hallucinations, by using them to design a lightweight hallucination detector.
  arXiv  Detail & Related papers  (2023-01-18T20:43:13Z)
• On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models? [32.41234580068662]
  We conduct a study on existing knowledge-grounded conversational benchmarks and several state-of-the-art models. Standard benchmarks consist of >60% hallucinated responses, leading to models that not only hallucinate but even amplify hallucinations. Our findings raise important questions on the quality of existing datasets and models trained using them.
  arXiv  Detail & Related papers  (2022-04-17T05:15:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.