Hallucinations Undermine Trust; Metacognition is a Way Forward

• URL: http://arxiv.org/abs/2605.01428v1
• Date: Sat, 02 May 2026 12:59:14 GMT
• Title: Hallucinations Undermine Trust; Metacognition is a Way Forward
• Abstract summary: Despite significant strides in factual reliability, errors -- often termed hallucinations -- remain a major concern for generative AI.<n>We argue that most factuality gains in this domain have come from expanding the model's knowledge boundary.<n>We propose faithful uncertainty: aligning linguistic uncertainty with intrinsic uncertainty.
• Score: 36.173804474959915
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite significant strides in factual reliability, errors -- often termed hallucinations -- remain a major concern for generative AI, especially as LLMs are increasingly expected to be helpful in more complex or nuanced setups. Yet even in the simplest setting -- factoid question-answering with clear ground truth-frontier models without external tools continue to hallucinate. We argue that most factuality gains in this domain have come from expanding the model's knowledge boundary (encoding more facts) rather than improving awareness of that boundary (distinguishing known from unknown). We conjecture that the latter is inherently difficult: models may lack the discriminative power to perfectly separate truths from errors, creating an unavoidable tradeoff between eliminating hallucinations and preserving utility. This tradeoff dissolves under a different framing. If we understand hallucinations as confident errors -- incorrect information delivered without appropriate qualification -- a third path emerges beyond the answer-or-abstain dichotomy: expressing uncertainty. We propose faithful uncertainty: aligning linguistic uncertainty with intrinsic uncertainty. This is one facet of metacognition -- the ability to be aware of one's own uncertainty and to act on it. For direct interaction, acting on uncertainty means communicating it honestly; for agentic systems, it becomes the control layer governing when to search and what to trust. Metacognition is thus essential for LLMs to be both trustworthy and capable; we conclude by highlighting open problems for progress towards this objective.

Related papers

• Reality Monitoring in Large Language Models: Self-Knowledge That Transforms with Conversation Memory [0.0]
  We show that source attribution depends on how conversational memory is structured.<n>This suggests that as AI systems take on autonomous, multi-turn roles, evaluating what they know is not enough.
  arXiv  Detail & Related papers  (2026-07-27T01:47:12Z)
• Detecting Misbehaviors of Large Vision-Language Models by Evidential Uncertainty Quantification [27.02252748004729]
  Large vision-language models (LVLMs) have shown substantial advances in multimodal understanding and generation.<n>They frequently produce unreliable or even harmful content, such as fact hallucinations or dangerous instructions.<n>Evidential Uncertainty Quantification (EUQ) captures both information conflict and ignorance for effective detection of LVLM misbehaviors.
  arXiv  Detail & Related papers  (2026-02-05T10:51:39Z)
• Two Pathways to Truthfulness: On the Intrinsic Encoding of LLM Hallucinations [70.43616821802249]
  Large language models (LLMs) frequently generate hallucinations.<n>Previous work shows that their internal states encode rich signals of truthfulness.<n>This paper shows that truthfulness cues arise from two distinct information pathways.
  arXiv  Detail & Related papers  (2026-01-12T11:10:43Z)
• HACK: Hallucinations Along Certainty and Knowledge Axes [66.66625343090743]
  We propose a framework for categorizing hallucinations along two axes: knowledge and certainty.<n>We identify a particularly concerning subset of hallucinations where models hallucinate with certainty despite having the correct knowledge internally.
  arXiv  Detail & Related papers  (2025-10-28T09:34:31Z)
• Beyond Hallucinations: The Illusion of Understanding in Large Language Models [0.0]
  Large language models (LLMs) are becoming deeply embedded in human communication and decision-making.<n>They inherit the ambiguity, bias, and lack of direct access to truth inherent in language itself.<n>This paper argues that LLMs operationalize System 1 cognition at scale: fast, associative, and persuasive, but without reflection or falsification.
  arXiv  Detail & Related papers  (2025-10-16T13:19:44Z)
• Auditing Meta-Cognitive Hallucinations in Reasoning Large Language Models [12.747507415841168]
  We study the causality of hallucinations under constrained knowledge domains by auditing the Chain-of-Thought trajectory.<n>Our analysis reveals that in long-CoT settings, RLLMs can iteratively reinforce biases and errors through flawed reflective reasoning.<n>Surprisingly, even direct interventions at the origin of hallucinations often fail to reverse their effects, as reasoning chains exhibit 'chain disloyalty'
  arXiv  Detail & Related papers  (2025-05-19T14:11:09Z)
• Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinations [51.92795774118647]
  We find that verbal uncertainty'' is governed by a single linear feature in the representation space of LLMs.<n>We show that this has only moderate correlation with the actual semantic uncertainty'' of the model.
  arXiv  Detail & Related papers  (2025-03-18T17:51:04Z)
• Delusions of Large Language Models [62.43923767408462]
  Large Language Models often generate factually incorrect but plausible outputs, known as hallucinations.<n>We identify a more insidious phenomenon, LLM delusion, defined as high belief hallucinations, incorrect outputs with abnormally high confidence, making them harder to detect and mitigate.
  arXiv  Detail & Related papers  (2025-03-09T17:59:16Z)
• Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answer [51.7407540261676]
  We investigate a distinct type of hallucination, where a model can consistently answer a question correctly, but a seemingly trivial perturbation causes it to produce a hallucinated response with high certainty.<n>This phenomenon is particularly concerning in high-stakes domains such as medicine or law, where model certainty is often used as a proxy for reliability.<n>We show that CHOKE examples are consistent across prompts, occur in different models and datasets, and are fundamentally distinct from other hallucinations.
  arXiv  Detail & Related papers  (2025-02-18T15:46:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.