Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation

• URL: http://arxiv.org/abs/2606.05403v1
• Date: Wed, 03 Jun 2026 20:15:48 GMT
• Title: Trust, but Don't Verify: Epistemic Blind Spots in LLM Source Evaluation
• Abstract summary: We show that models possess the capability to detect fabricated statistics but do not recruit this capability during multi-source synthesis.<n>Specifically, source influence is governed by a methodology-register gate that responds to the distributional register of analytical text but not to numeric validity.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models increasingly act as epistemic proxies, synthesizing evidence from multiple sources to inform decisions. Whether they evaluate the quality of that evidence, or merely aggregate it based on surface presentation, remains poorly understood. We show that models possess the capability to detect fabricated statistics (correct identification rates of 0.76-1.00 for methodology in isolation) but do not recruit this capability during multi-source synthesis, producing similar numeric estimates whether the statistics are fabricated or valid. Specifically, source influence is governed by a methodology-register gate that responds to the distributional register of analytical text but not to numeric validity: for example, statistically impossible confidence intervals receive the same weight as valid ones. The behavioral dissociation replicates across five models from three families (Claude, Qwen, OLMo) and three professional domains. Mechanistic analyses, including causal tracing, linear probes, and component-level attribution, converge on the same account: the model encodes and causally uses a methodology-register representation that transfers across domains (probe AUC 0.83-0.92), while numeric-validity signals, decodable in isolation, are suppressed to chance during multi-source synthesis. Prompting-based mitigations, even an oracle checklist naming the exact statistical checks, produce blanket skepticism rather than selective discernment, and the post-training pipelines we examine reinforce the stylistic shortcut without building numeric verification. Unlike sycophancy, which tracks user preference, this failure tracks whether a source presents as analytically credible, not whether its claims are internally consistent. We term this epistemic alignment: like preference and safety alignment, the question is not capability but deployment.

Related papers

• TelemetrySuffBench: Is Agent Telemetry Sufficient for Failure-Origin Diagnosis? [1.7077233340294056]
  We introduce TelemetrySuffBench, a benchmark that separates failure detection, fault-origin localization, and safe abstention under insufficient evidence.<n>We evaluate five frontier language models using unified protocols, explicit candidate sets, invalid-output accounting, subgroup analyses, and a frozen blind holdout.
  arXiv  Detail & Related papers  (2026-08-08T03:54:22Z)
• The Digital Twin Counterfactual Framework: A Validation Architecture for Simulated Potential Outcomes [0.0]
  Digital Twin Counterfactual Framework (DTCF)<n>Five-level validation architecture converts unfalsifiable claim that simulator produces correct counterfactuals into falsifiable tests against observable data.<n> decomposition separates causal quantities into those that are marginally validated.<n>Joint causal claims become explicitly assumption-indexed.
  arXiv  Detail & Related papers  (2026-04-01T19:04:15Z)
• When Stability Fails: Hidden Failure Modes Of LLMS in Data-Constrained Scientific Decision-Making [0.0]
  Large language models (LLMs) are increasingly used as decision-support tools in data-constrained scientific work- flows.<n>We introduce a controlled behavioral evaluation framework that explicitly seprates four dimensions of LLM decision-making.<n>We show that LLMs can exhibit near-perfect run-to-run stability while diverging from statistical ground truth.
  arXiv  Detail & Related papers  (2026-03-16T19:17:09Z)
• Statistical Inference via Generative Models: Flow Matching and Causal Inference [0.0]
  Generative AI has achieved remarkable empirical success, but from the perspective of statistics it often remains opaque.<n>This book reinterprets generative AI in the language of statistics, using flow matching as a central example.
  arXiv  Detail & Related papers  (2026-03-09T22:56:02Z)
• Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulness [30.401369966789915]
  Pass@k and other methods of scaling inference compute can improve language model performance in domains with external verifiers.<n>We show that across five benchmarks and models, surprisingly, it cannot.<n>We find that under uncertainty, models are better at predicting what other models will say within model ensembles than at identifying what is true.
  arXiv  Detail & Related papers  (2026-02-20T03:35:01Z)
• Large Language Models Are Bad Dice Players: LLMs Struggle to Generate Random Numbers from Statistical Distributions [50.1404916337174]
  We present the first large-scale, statistically powered audit of native probabilistic sampling in large language models (LLMs)<n>We show that batch generation achieves only modest statistical validity, with a 13% median pass rate, while independent requests collapse almost entirely.<n>We conclude that current LLMs lack a functional internal sampler, necessitating the use of external tools for applications requiring statistical guarantees.
  arXiv  Detail & Related papers  (2026-01-08T22:33:12Z)
• Statistical Hypothesis Testing for Auditing Robustness in Language Models [49.1574468325115]
  We introduce distribution-based perturbation analysis, a framework that reformulates perturbation analysis as a frequentist hypothesis testing problem.<n>We construct empirical null and alternative output distributions within a low-dimensional semantic similarity space via Monte Carlo sampling.<n>We show how we can quantify response changes, measure true/false positive rates, and evaluate alignment with reference models.
  arXiv  Detail & Related papers  (2025-06-09T17:11:07Z)
• Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling [69.83976050879318]
  In large language models (LLMs), identifying sources of uncertainty is an important step toward improving reliability, trustworthiness, and interpretability. In this paper, we introduce an uncertainty decomposition framework for LLMs, called input clarification ensembling. Our approach generates a set of clarifications for the input, feeds them into an LLM, and ensembles the corresponding predictions.
  arXiv  Detail & Related papers  (2023-11-15T05:58:35Z)
• Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval [139.21955930418815]
  Cross-modal Retrieval methods build similarity relations between vision and language modalities by jointly learning a common representation space. However, the predictions are often unreliable due to the Aleatoric uncertainty, which is induced by low-quality data, e.g., corrupt images, fast-paced videos, and non-detailed texts. We propose a novel Prototype-based Aleatoric Uncertainty Quantification (PAU) framework to provide trustworthy predictions by quantifying the uncertainty arisen from the inherent data ambiguity.
  arXiv  Detail & Related papers  (2023-09-29T09:41:19Z)
• Causal Discovery via Conditional Independence Testing with Proxy Variables [35.3493980628004]
  The presence of unobserved variables, such as the latent confounder, can introduce bias in conditional independence testing. We propose a novel hypothesis-testing procedure that can effectively examine the existence of the causal relationship over continuous variables.
  arXiv  Detail & Related papers  (2023-05-09T09:08:39Z)
• Tracking the risk of a deployed model and detecting harmful distribution shifts [105.27463615756733]
  In practice, it may make sense to ignore benign shifts, under which the performance of a deployed model does not degrade substantially. We argue that a sensible method for firing off a warning has to both (a) detect harmful shifts while ignoring benign ones, and (b) allow continuous monitoring of model performance without increasing the false alarm rate.
  arXiv  Detail & Related papers  (2021-10-12T17:21:41Z)
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning [123.3472310767721]
  Prediction credibility measures are fundamental in statistics and machine learning. These measures should account for the wide variety of models used in practice. The framework developed in this work expresses the credibility as a risk-fit trade-off.
  arXiv  Detail & Related papers  (2020-11-24T19:52:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.