Hallucination is Inevitable for LLMs with the Open World Assumption

• URL: http://arxiv.org/abs/2510.05116v1
• Date: Mon, 29 Sep 2025 13:38:44 GMT
• Title: Hallucination is Inevitable for LLMs with the Open World Assumption
• Authors: Bowen Xu,
• Abstract summary: Large Language Models (LLMs) exhibit impressive linguistic competence but also produce inaccurate or fabricated outputs, often called hallucinations''<n>This paper reframes hallucination'' as a manifestation of the generalization problem.
• Score: 10.473344768196908
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) exhibit impressive linguistic competence but also produce inaccurate or fabricated outputs, often called ``hallucinations''. Engineering approaches usually regard hallucination as a defect to be minimized, while formal analyses have argued for its theoretical inevitability. Yet both perspectives remain incomplete when considering the conditions required for artificial general intelligence (AGI). This paper reframes ``hallucination'' as a manifestation of the generalization problem. Under the Closed World assumption, where training and test distributions are consistent, hallucinations may be mitigated. Under the Open World assumption, however, where the environment is unbounded, hallucinations become inevitable. This paper further develops a classification of hallucination, distinguishing cases that may be corrected from those that appear unavoidable under open-world conditions. On this basis, it suggests that ``hallucination'' should be approached not merely as an engineering defect but as a structural feature to be tolerated and made compatible with human intelligence.

Related papers

• Incentives or Ontology? A Structural Rebuttal to OpenAI's Hallucination Thesis [0.42970700836450487]
  We argue that hallucination is not an optimization failure but an architectural inevitability of the transformer model.<n>Our empirical results demonstrate that hallucination can only be eliminated through external truth-validation and abstention modules.<n>We conclude that hallucination is a structural property of generative architectures.
  arXiv  Detail & Related papers  (2025-12-16T17:39:45Z)
• Review of Hallucination Understanding in Large Language and Vision Models [65.29139004945712]
  We present a framework for characterizing both image and text hallucinations across diverse applications.<n>Our investigations reveal that hallucinations often stem from predictable patterns in data distributions and inherited biases.<n>This survey provides a foundation for developing more robust and effective solutions to hallucinations in real-world generative AI systems.
  arXiv  Detail & Related papers  (2025-09-26T09:23:08Z)
• How Large Language Models are Designed to Hallucinate [0.42970700836450487]
  We argue that hallucination is a structural outcome of the transformer architecture.<n>Our contribution is threefold: (1) a comparative account showing why existing explanations are insufficient; (2) a predictive taxonomy of hallucination linked to existential structures with proposed benchmarks; and (3) design directions toward "truth-constrained" architectures capable of withholding or deferring when disclosure is absent.
  arXiv  Detail & Related papers  (2025-09-19T16:46:27Z)
• Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer [51.7407540261676]
  We investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation causes it to produce a hallucinated response with high certainty.<n>This phenomenon is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability.<n>We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations.
  arXiv  Detail & Related papers  (2025-02-18T15:46:31Z)
• Valuable Hallucinations: Realizable Non-realistic Propositions [2.451326684641447]
  This paper introduces the first formal definition of valuable hallucinations in large language models (LLMs)<n>We focus on the potential value that certain types of hallucinations can offer in specific contexts.<n>We present experiments using the Qwen2.5 model and HalluQA dataset, employing ReAct prompting to control and optimize hallucinations.
  arXiv  Detail & Related papers  (2025-02-16T12:59:11Z)
• LLMs Will Always Hallucinate, and We Need to Live With This [1.3810901729134184]
  This work argues that hallucinations in language models are not just occasional errors but an inevitable feature of these systems. It is, therefore, impossible to eliminate them through architectural improvements, dataset enhancements, or fact-checking mechanisms.
  arXiv  Detail & Related papers  (2024-09-09T16:01:58Z)
• Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Models [65.32990889402927]
  We coin this phenomenon as knowledge overshadowing'' We show that the hallucination rate grows with both the imbalance ratio and the length of dominant condition description. We propose to utilize overshadowing conditions as a signal to catch hallucination before it is produced.
  arXiv  Detail & Related papers  (2024-07-10T20:37:42Z)
• On Large Language Models' Hallucination with Regard to Known Facts [74.96789694959894]
  Large language models are successful in answering factoid questions but are also prone to hallucination. We investigate the phenomenon of LLMs possessing correct answer knowledge yet still hallucinating from the perspective of inference dynamics. Our study shed light on understanding the reasons for LLMs' hallucinations on their known facts, and more importantly, on accurately predicting when they are hallucinating.
  arXiv  Detail & Related papers  (2024-03-29T06:48:30Z)
• Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations [42.46721214112836]
  State-of-the-art language models (LMs) sometimes generate non-factual hallucinations that misalign with world knowledge. We create diagnostic datasets with subject-relation queries and adapt interpretability methods to trace hallucinations through internal model representations.
  arXiv  Detail & Related papers  (2024-03-27T00:23:03Z)
• Hallucination is Inevitable: An Innate Limitation of Large Language Models [3.4444349898613957]
  We show that it is impossible to eliminate hallucination in large language models.<n>Since the formal world is a part of the real world which is much more complicated, hallucinations are also inevitable for real world LLMs.
  arXiv  Detail & Related papers  (2024-01-22T10:26:14Z)
• Do Androids Know They're Only Dreaming of Electric Sheep? [45.513432353811474]
  We design probes trained on the internal representations of a transformer language model to predict its hallucinatory behavior. Our probes are narrowly trained and we find that they are sensitive to their training domain. We find that probing is a feasible and efficient alternative to language model hallucination evaluation when model states are available.
  arXiv  Detail & Related papers  (2023-12-28T18:59:50Z)
• Probing Causes of Hallucinations in Neural Machine Translations [51.418245676894465]
  We propose to use probing methods to investigate the causes of hallucinations from the perspective of model architecture. We find that hallucination is often accompanied by the deficient encoder, especially embeddings, and vulnerable cross-attentions.
  arXiv  Detail & Related papers  (2022-06-25T01:57:22Z)
• On Hallucination and Predictive Uncertainty in Conditional Language Generation [76.18783678114325]
  Higher predictive uncertainty corresponds to a higher chance of hallucination. Epistemic uncertainty is more indicative of hallucination than aleatoric or total uncertainties. It helps to achieve better results of trading performance in standard metric for less hallucination with the proposed beam search variant.
  arXiv  Detail & Related papers  (2021-03-28T00:32:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.