DECK: A Consistency x Confidence Taxonomy of LLM Hallucinations

• URL: http://arxiv.org/abs/2606.02289v1
• Date: Mon, 01 Jun 2026 14:11:11 GMT
• Title: DECK: A Consistency x Confidence Taxonomy of LLM Hallucinations
• Abstract summary: Existing hallucination classify errors by what is wrong with the output -- memorised misconceptions, reasoning failures, fluent fabrications.<n>These are useful for diagnosis but cannot answer a different question: which uncertainty scorer would have caught this error?<n>We propose a complementary taxonomy that classifies errors by their detectability signature -- the signal a scorer family would read.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing hallucination taxonomies classify LLM errors by what is wrong with the output -- memorised misconceptions, reasoning failures, fluent fabrications. These taxonomies are useful for diagnosis but cannot answer a different question: which uncertainty scorer would have caught this error? We propose a complementary taxonomy that classifies errors by their detectability signature -- the signal a scorer family would read. The DECK taxonomy is a 2x2 partition along inter-sample consistency and token-level confidence into four behavioural regimes (Drift, Entrenched, Confabulation, Knotted), each mapping to a specific scorer family (or families) that can detect it: black-box consistency scorers have signal in D and C, white-box token-probability scorers have signal in K and C, and only an LLM-as-a-Judge with independent pretraining can detect E. Cell membership is operationalised by a Youden's J optimal split on each scorer axis. Across three models and four datasets we validate the taxonomy two ways: by analysing scorer-pair disagreement, and by checking that external labels (SelfAware unanswerable, HaluEval adversarial, PopQA entity popularity) land in the predicted DECK cells, with model-scale and content-specific secondary-cell refinements. We further identify a universal blind spot of output-level UQ: on knowledge-gap inputs where the generator emits confident, repeatable fabrications, every output-level family collapses by construction. A linear probe on Llama-3-8B's hidden states also collapses to chance, giving preliminary evidence that the failure may persist at the activation level; richer internal-state methods (UQ heads, information-theoretic estimators) remain to be tested.

Related papers

• From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs [0.9751538234290761]
  Missing fabrications can harm users and institutions, while false alarms consume limited human-review capacity.<n>We study two signals accessible through black-box model APIs: semantic entropy, and uncertainty derived from token log-probabilities.<n>We extend token-based uncertainty detection by aggregating token-level signals across sampled responses through our TopK method.
  arXiv  Detail & Related papers  (2026-09-02T14:50:24Z)
• Grading the Graders: Verification Autonomy Levels (L0-L5) for LLM Reasoning [0.0]
  We propose Verification Autonomy Levels (VAL), a meta-standard that classifies any verification scheme along a single axis.<n>VAL ranges from L0 (LLM self-declaration; no deterministic anchor) through L2 (objective ground truth; correctness only) to L3 (decidable systems with single-property or domain-level completeness)<n>Central to VAL is the completeness blind spot: substitution- and sampling-based verifiers can confirm that proposed candidates hold, but cannot prove that no candidate was missed.
  arXiv  Detail & Related papers  (2026-08-19T15:10:00Z)
• Intrinsic Structure: Spectral Identifiability for Mechanistic Interpretability [51.56484100374058]
  We prove the first identifiability theorem for a mechanistic-interpretability primitive.<n>The spectrum is an identifiable, model-intrinsic fingerprint with a stated error bar, not a legible decomposition.
  arXiv  Detail & Related papers  (2026-08-10T19:42:01Z)
• Knowledge before Reasoning: EC-Reason-Bench, a Training-Free Diagnostic Benchmark for LLM Enzyme Classification [36.44850004100564]
  We propose EC-Reason-Bench, a training-free, diagnostic evaluation protocol built to answer two questions.<n>First, external knowledge is decisive and must precede reasoning.<n>Second, in closed-book settings, whether cascading and chain-of-thought help or hurt depends on a model's tendency to abstain.
  arXiv  Detail & Related papers  (2026-07-29T02:16:09Z)
• Deterministic Integrity Gates for LLM-Assisted Clinical Manuscript Preparation: An Auditable Biomedical Informatics Architecture [0.3823356975862005]
  We describe an architecture pairing generation with verification, resting on three principles.<n>This determinism-where-possible split, organized as an integrity-gate taxonomy, is the core contribution.<n> MedSci Skills is an open-source toolkit of 43 skills with a 21-detector deterministic tier, evaluated on three public-dataset pipelines.
  arXiv  Detail & Related papers  (2026-06-08T13:51:04Z)
• Decoding in Order-Agnostic Language Models: Chain-Rule Deviation and Uniform Spreading [0.6916773850242582]
  Order-agnostic language models (OALMs) are trained to predict masked tokens under arbitrary conditioning sets.<n>We show that learning conditionals are not exact factorizations of a coherent joint distribution.<n>We propose a complementary diagnostic based on the shape of the confidence trace.
  arXiv  Detail & Related papers  (2026-05-31T04:25:36Z)
• How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signals [6.467495925520036]
  Large language models can detect their own errors and sometimes correct them without external feedback.<n>We investigate this through the lens of second-order models of confidence from decision neuroscience.
  arXiv  Detail & Related papers  (2026-04-24T06:33:32Z)
• Eidoku: A Neuro-Symbolic Verification Gate for LLM Reasoning via Structural Constraint Satisfaction [0.0]
  Large Language Models (LLMs) frequently produce hallucinated statements that are assigned high likelihood by the model itself.<n>This suggests that hallucination is often not a low-confidence phenomenon, but a failure of structural consistency.<n>We reformulate the verification of LLM reasoning as a Constraint Satisfaction Problem (CSP) operating independently of the generation likelihood.
  arXiv  Detail & Related papers  (2025-12-19T05:29:43Z)
• CLUE: Non-parametric Verification from Experience via Hidden-State Clustering [64.50919789875233]
  We show that correctness of a solution is encoded as a geometrically separable signature within the trajectory of hidden activations.<n>ClUE consistently outperforms LLM-as-a-judge baselines and matches or exceeds modern confidence-based methods in reranking candidates.
  arXiv  Detail & Related papers  (2025-10-02T02:14:33Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Unsupervised Conformal Inference: Bootstrapping and Alignment to Control LLM Uncertainty [49.19257648205146]
  We propose an unsupervised conformal inference framework for generation.<n>Our gates achieve close-to-nominal coverage and provide tighter, more stable thresholds than split UCP.<n>The result is a label-free, API-compatible gate for test-time filtering.
  arXiv  Detail & Related papers  (2025-09-26T23:40:47Z)
• Detecting Token-Level Hallucinations Using Variance Signals: A Reference-Free Approach [0.0]
  Large Language Models (LLMs) have demonstrated impressive generative capabilities across diverse tasks but remain susceptible to hallucinations.<n>We introduce a reference-free, token-level hallucination detection framework that leverages the variance in token log-probabilities across multiple generations.<n>Our approach is model-agnostic, interpretable, and suited for real-time or post-hoc analysis.
  arXiv  Detail & Related papers  (2025-07-05T19:20:59Z)
• DeCaFlow: A deconfounding causal generative model [48.86777554649186]
  We introduce DeCaFlow, a deconfounding causal generative model.<n>We extend previous results on causal estimation under hidden confounding to show that a single instance of DeCaFlow provides correct estimates for all causal queries identifiable with do-calculus.<n>Our empirical results on diverse settings show that DeCaFlow outperforms existing approaches, while demonstrating its out-of-the-box applicability to any given causal graph.
  arXiv  Detail & Related papers  (2025-03-19T11:14:16Z)
• Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification [116.77055746066375]
  Large language models (LLMs) are notorious for hallucinating, i.e., producing erroneous claims in their output. We propose a novel fact-checking and hallucination detection pipeline based on token-level uncertainty quantification.
  arXiv  Detail & Related papers  (2024-03-07T17:44:17Z)
• Nested Counterfactual Identification from Arbitrary Surrogate Experiments [95.48089725859298]
  We study the identification of nested counterfactuals from an arbitrary combination of observations and experiments. Specifically, we prove the counterfactual unnesting theorem (CUT), which allows one to map arbitrary nested counterfactuals to unnested ones.
  arXiv  Detail & Related papers  (2021-07-07T12:51:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.