Hallucination, reliability, and the role of generative AI in science

• URL: http://arxiv.org/abs/2504.08526v1
• Date: Fri, 11 Apr 2025 13:38:56 GMT
• Title: Hallucination, reliability, and the role of generative AI in science
• Authors: Charles Rathkopf,
• Abstract summary: Some arguments suggest that hallucinations are an inevitable consequence of the mechanisms underlying generative inference.<n>I argue that although corrosive hallucinations do pose a threat to scientific reliability, they are not inevitable.
• Score: 0.05657375260432172
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Generative AI is increasingly used in scientific domains, from protein folding to climate modeling. But these models produce distinctive errors known as hallucinations - outputs that are incorrect yet superficially plausible. Worse, some arguments suggest that hallucinations are an inevitable consequence of the mechanisms underlying generative inference. Fortunately, such arguments rely on a conception of hallucination defined solely with respect to internal properties of the model, rather than in reference to the empirical target system. This conception fails to distinguish epistemically benign errors from those that threaten scientific inference. I introduce the concept of corrosive hallucination to capture the epistemically troubling subclass: misrepresentations that are substantively misleading and resistant to systematic anticipation. I argue that although corrosive hallucinations do pose a threat to scientific reliability, they are not inevitable. Scientific workflows such as those surrounding AlphaFold and GenCast, both of which serve as case studies, can neutralize their effects by imposing theoretical constraints during training, and by strategically screening for errors at inference time. When embedded in such workflows, generative AI can reliably contribute to scientific knowledge.

Related papers

• Incentives or Ontology? A Structural Rebuttal to OpenAI's Hallucination Thesis [0.42970700836450487]
  We argue that hallucination is not an optimization failure but an architectural inevitability of the transformer model.<n>Our empirical results demonstrate that hallucination can only be eliminated through external truth-validation and abstention modules.<n>We conclude that hallucination is a structural property of generative architectures.
  arXiv  Detail & Related papers  (2025-12-16T17:39:45Z)
• HACK: Hallucinations Along Certainty and Knowledge Axes [66.66625343090743]
  We propose a framework for categorizing hallucinations along two axes: knowledge and certainty.<n>We identify a particularly concerning subset of hallucinations where models hallucinate with certainty despite having the correct knowledge internally.
  arXiv  Detail & Related papers  (2025-10-28T09:34:31Z)
• Distributional Semantics Tracing: A Framework for Explaining Hallucinations in Large Language Models [4.946483489399819]
  Large Language Models (LLMs) are prone to hallucination, the generation of factually incorrect statements.<n>This work investigates the intrinsic, architectural origins of this failure mode through three primary contributions.
  arXiv  Detail & Related papers  (2025-10-07T16:40:31Z)
• Review of Hallucination Understanding in Large Language and Vision Models [65.29139004945712]
  We present a framework for characterizing both image and text hallucinations across diverse applications.<n>Our investigations reveal that hallucinations often stem from predictable patterns in data distributions and inherited biases.<n>This survey provides a foundation for developing more robust and effective solutions to hallucinations in real-world generative AI systems.
  arXiv  Detail & Related papers  (2025-09-26T09:23:08Z)
• How Large Language Models are Designed to Hallucinate [0.42970700836450487]
  We argue that hallucination is a structural outcome of the transformer architecture.<n>Our contribution is threefold: (1) a comparative account showing why existing explanations are insufficient; (2) a predictive taxonomy of hallucination linked to existential structures with proposed benchmarks; and (3) design directions toward "truth-constrained" architectures capable of withholding or deferring when disclosure is absent.
  arXiv  Detail & Related papers  (2025-09-19T16:46:27Z)
• Active Inference AI Systems for Scientific Discovery [1.450405446885067]
  This perspective contends that progress turns on closing three mutually reinforcing gaps in abstraction, reasoning and empirical grounding.<n>Design principles are proposed for systems that reason in imaginary spaces and learn from the world.
  arXiv  Detail & Related papers  (2025-06-26T14:43:04Z)
• Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models [8.97308732968526]
  We study the causality of hallucinations under constrained knowledge domains by auditing the Chain-of-Thought trajectory.<n>Our analysis reveals that in long-CoT settings, RLLMs can iteratively reinforce biases and errors through flawed reflective reasoning.<n>Surprisingly, even direct interventions at the origin of hallucinations often fail to reverse their effects.
  arXiv  Detail & Related papers  (2025-05-19T14:11:09Z)
• Beyond Misinformation: A Conceptual Framework for Studying AI Hallucinations in (Science) Communication [0.0]
  This paper proposes a conceptual framework for understanding AI hallucinations as a distinct form of misinformation.<n>I argue that these AI hallucinations should not be treated merely as technical failures but as communication phenomena with social consequences.
  arXiv  Detail & Related papers  (2025-04-18T16:26:02Z)
• Delusions of Large Language Models [62.43923767408462]
  Large Language Models often generate factually incorrect but plausible outputs, known as hallucinations. We identify a more insidious phenomenon, LLM delusion, defined as high belief hallucinations, incorrect outputs with abnormally high confidence, making them harder to detect and mitigate.
  arXiv  Detail & Related papers  (2025-03-09T17:59:16Z)
• The Law of Knowledge Overshadowing: Towards Understanding, Predicting, and Preventing LLM Hallucination [85.18584652829799]
  We introduce a novel framework to quantify factual hallucinations by modeling knowledge overshadowing.<n>We propose a new decoding strategy CoDa, to mitigate hallucinations, which notably enhance model factuality on Overshadow (27.9%), MemoTrap (13.1%) and NQ-Swap (18.3%)
  arXiv  Detail & Related papers  (2025-02-22T08:36:06Z)
• Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer [51.7407540261676]
  We investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation causes it to produce a hallucinated response with high certainty.<n>This phenomenon is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability.<n>We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations.
  arXiv  Detail & Related papers  (2025-02-18T15:46:31Z)
• Hallucinations are inevitable but can be made statistically negligible. The "innate" inevitability of hallucinations cannot explain practical LLM issues [23.12905996928255]
  Hallucinations, a phenomenon where a language model (LM) generates nonfactual content, pose a significant challenge to the practical deployment of LMs.<n>Recent studies established a computability-theoretic result showing that any LM will inevitably generate hallucinations on an infinite set of inputs.<n>This paper claims that those "innate" inevitability results from computability theory and diagonal argument, in principle, cannot explain practical issues of LMs.
  arXiv  Detail & Related papers  (2025-02-15T07:28:40Z)
• Generative Intervention Models for Causal Perturbation Modeling [80.72074987374141]
  In many applications, it is a priori unknown which mechanisms of a system are modified by an external perturbation. We propose a generative intervention model (GIM) that learns to map these perturbation features to distributions over atomic interventions.
  arXiv  Detail & Related papers  (2024-11-21T10:37:57Z)
• No Free Lunch: Fundamental Limits of Learning Non-Hallucinating Generative Models [14.535583931446807]
  We develop a theoretical framework to analyze the learnability of non-hallucinating generative models. We show that incorporating inductive biases aligned with the actual facts into the learning process is essential.
  arXiv  Detail & Related papers  (2024-10-24T23:57:11Z)
• Do Robot Snakes Dream like Electric Sheep? Investigating the Effects of Architectural Inductive Biases on Hallucination [41.70182291774456]
  Large language models (LLMs) tend to hallucinate false or misleading information, limiting their reliability.<n>Do changes in architecture exacerbate/alleviate existing concerns about hallucinations?<n>These findings highlight the need for better understanding both these problems in conjunction with each other, as well as consider how to design more universal techniques for handling hallucinations.
  arXiv  Detail & Related papers  (2024-10-22T23:24:15Z)
• LLMs Will Always Hallucinate, and We Need to Live With This [1.3810901729134184]
  This work argues that hallucinations in language models are not just occasional errors but an inevitable feature of these systems. It is, therefore, impossible to eliminate them through architectural improvements, dataset enhancements, or fact-checking mechanisms.
  arXiv  Detail & Related papers  (2024-09-09T16:01:58Z)
• Smoke and Mirrors in Causal Downstream Tasks [59.90654397037007]
  This paper looks at the causal inference task of treatment effect estimation, where the outcome of interest is recorded in high-dimensional observations.<n>We compare 6 480 models fine-tuned from state-of-the-art visual backbones, and find that the sampling and modeling choices significantly affect the accuracy of the causal estimate.<n>Our results suggest that future benchmarks should carefully consider real downstream scientific questions, especially causal ones.
  arXiv  Detail & Related papers  (2024-05-27T13:26:34Z)
• Looks Too Good To Be True: An Information-Theoretic Analysis of Hallucinations in Generative Restoration Models [13.605340325383452]
  generative models are capable of producing results often visually indistinguishable from real data. They also exhibit a growing tendency to generate hallucinations - realistic-looking details that do not exist in the ground truth images. This paper investigates this phenomenon through the lens of information theory, revealing a fundamental tradeoff between uncertainty and perception.
  arXiv  Detail & Related papers  (2024-05-26T07:58:51Z)
• A Cause-Effect Look at Alleviating Hallucination of Knowledge-grounded Dialogue Generation [51.53917938874146]
  We propose a possible solution for alleviating the hallucination in KGD by exploiting the dialogue-knowledge interaction. Experimental results of our example implementation show that this method can reduce hallucination without disrupting other dialogue performance.
  arXiv  Detail & Related papers  (2024-04-04T14:45:26Z)
• On Large Language Models' Hallucination with Regard to Known Facts [74.96789694959894]
  Large language models are successful in answering factoid questions but are also prone to hallucination. We investigate the phenomenon of LLMs possessing correct answer knowledge yet still hallucinating from the perspective of inference dynamics. Our study shed light on understanding the reasons for LLMs' hallucinations on their known facts, and more importantly, on accurately predicting when they are hallucinating.
  arXiv  Detail & Related papers  (2024-03-29T06:48:30Z)
• ACRE: Abstract Causal REasoning Beyond Covariation [90.99059920286484]
  We introduce the Abstract Causal REasoning dataset for systematic evaluation of current vision systems in causal induction. Motivated by the stream of research on causal discovery in Blicket experiments, we query a visual reasoning system with the following four types of questions in either an independent scenario or an interventional scenario. We notice that pure neural models tend towards an associative strategy under their chance-level performance, whereas neuro-symbolic combinations struggle in backward-blocking reasoning.
  arXiv  Detail & Related papers  (2021-03-26T02:42:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.