Causal Understanding by LLMs: The Role of Uncertainty

• URL: http://arxiv.org/abs/2509.20088v1
• Date: Wed, 24 Sep 2025 13:06:35 GMT
• Title: Causal Understanding by LLMs: The Role of Uncertainty
• Abstract summary: Recent papers show LLMs achieve near-random accuracy in causal relation classification.<n>We investigate whether pretraining exposure to causal examples improves causal understanding.
• Score: 43.87879175532034
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent papers show LLMs achieve near-random accuracy in causal relation classification, raising questions about whether such failures arise from limited pretraining exposure or deeper representational gaps. We investigate this under uncertainty-based evaluation, testing whether pretraining exposure to causal examples improves causal understanding >18K PubMed sentences -- half from The Pile corpus, half post-2024 -- across seven models (Pythia-1.4B/7B/12B, GPT-J-6B, Dolly-7B/12B, Qwen-7B). We analyze model behavior through: (i) causal classification, where the model identifies causal relationships in text, and (ii) verbatim memorization probing, where we assess whether the model prefers previously seen causal statements over their paraphrases. Models perform four-way classification (direct/conditional/correlational/no-relationship) and select between originals and their generated paraphrases. Results show almost identical accuracy on seen/unseen sentences (p > 0.05), no memorization bias (24.8% original selection), and output distribution over the possible options is almost flat, with entropic values near the maximum (1.35/1.39), confirming random guessing. Instruction-tuned models show severe miscalibration (Qwen: > 95% confidence, 32.8% accuracy, ECE=0.49). Conditional relations induce highest entropy (+11% vs. direct). These findings suggest that failures in causal understanding arise from the lack of structured causal representation, rather than insufficient exposure to causal examples during pretraining.

Related papers

• From Causal Plausibility to Causal Reliability: Evaluating LLMs as Calibrated Direct Causal-Edge Classifiers [15.44881677475537]
  Large language models (LLMs) are increasingly used to provide prior causal knowledge for structural causal discovery.<n>We evaluate 12 instruction-tuned open-weight models across six benchmark causal graphs, five prompting strategies, and four confidence sources.<n>Our results suggest LLMs are better viewed as sources of externally validated soft causal priors than as direct evidence of causal structure.
  arXiv  Detail & Related papers  (2026-08-24T15:55:47Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Verbal Confidence Saturation in 3-9B Open-Weight Instruction-Tuned LLMs: A Pre-Registered Psychometric Validity Screen [0.0]
  We tested whether seven instruction-tuned open-weight models produce verbalised confidence that meets minimal validity criteria.<n>We found Categorical elicitation did not rescue validity.<n> Token-level logprobability did not usefully predict verbalised confidence under the observed variance regime.
  arXiv  Detail & Related papers  (2026-04-24T04:45:21Z)
• Product-of-Experts Training Reduces Dataset Artifacts in Natural Language Inference [0.0]
  We propose Product-of-Experts (PoE) training, which downweights examples where biased models are overconfident.<n>PoE nearly preserves accuracy (89.10% vs. 89.30%) while cutting bias by 4.71% (bias agreement 49.85% to 45%)
  arXiv  Detail & Related papers  (2026-04-21T04:23:20Z)
• The Model Says Walk: How Surface Heuristics Override Implicit Constraints in LLM Reasoning [9.898274894485107]
  Large language models systematically fail when a salient surface cue conflicts with an unstated feasibility constraint.<n>We study this through a diagnose-measure-bridge-treat framework.
  arXiv  Detail & Related papers  (2026-03-30T21:36:09Z)
• Evaluating and Enhancing the Vulnerability Reasoning Capabilities of Large Language Models [15.849480549367684]
  We propose DAGVul, a novel framework that models vulnerability reasoning as a Directed Acyclic Graph (DAG) generation task.<n>By further introducing Reinforcement Learning with Verifiable Rewards (RLVR), we align model reasoning trace with program-intrinsic logic.<n>Our framework improves the reasoning F1-score by an average of 18.9% over all the baselines.
  arXiv  Detail & Related papers  (2026-02-06T13:19:45Z)
• CLEAR-3K: Assessing Causal Explanatory Capabilities in Language Models [3.137688620241855]
  We introduce CLEAR-3K, a dataset of 3,000 assertion-reasoning questions designed to evaluate whether language models can determine if one statement causally explains another.<n>Each question present an assertion-reason pair and challenge language models to distinguish between semantic relatedness and genuine causal explanatory relationships.
  arXiv  Detail & Related papers  (2025-06-20T17:35:36Z)
• Improving Group Robustness on Spurious Correlation via Evidential Alignment [26.544938760265136]
  Deep neural networks often learn and rely on spurious correlations, i.e., superficial associations between non-causal features and the targets.<n>Existing methods typically mitigate this issue by using external group annotations or auxiliary deterministic models.<n>We propose Evidential Alignment, a novel framework that leverages uncertainty quantification to understand the behavior of the biased models.
  arXiv  Detail & Related papers  (2025-06-12T22:47:21Z)
• A Closer Look at Bias and Chain-of-Thought Faithfulness of Large (Vision) Language Models [53.18562650350898]
  Chain-of-thought (CoT) reasoning enhances performance of large language models.<n>We present the first comprehensive study of CoT faithfulness in large vision-language models.
  arXiv  Detail & Related papers  (2025-05-29T18:55:05Z)
• Do Large Language Models Show Biases in Causal Learning? [3.0264418764647605]
  Causal learning is the cognitive process of developing the capability of making causal inferences based on available information.<n>This research investigates whether large language models (LLMs) develop causal illusions.
  arXiv  Detail & Related papers  (2024-12-13T19:03:48Z)
• Ensembled Prediction Intervals for Causal Outcomes Under Hidden Confounding [49.1865229301561]
  We present a simple approach to partial identification using existing causal sensitivity models and show empirically that Caus-Modens gives tighter outcome intervals. The last of our three diverse benchmarks is a novel usage of GPT-4 for observational experiments with unknown but probeable ground truth.
  arXiv  Detail & Related papers  (2023-06-15T21:42:40Z)
• Nonparametric Identifiability of Causal Representations from Unknown Interventions [63.1354734978244]
  We study causal representation learning, the task of inferring latent causal variables and their causal relations from mixtures of the variables. Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data.
  arXiv  Detail & Related papers  (2023-06-01T10:51:58Z)
• Cause-Effect Inference in Location-Scale Noise Models: Maximum Likelihood vs. Independence Testing [19.23479356810746]
  A fundamental problem of causal discovery is cause-effect inference, learning the correct causal direction between two random variables. Recently introduced heteroscedastic location-scale noise functional models (LSNMs) combine expressive power with identifiability guarantees. We show that LSNM model selection based on maximizing likelihood achieves state-of-the-art accuracy, when the noise distributions are correctly specified.
  arXiv  Detail & Related papers  (2023-01-26T20:48:32Z)
• Active Bayesian Causal Inference [72.70593653185078]
  We propose Active Bayesian Causal Inference (ABCI), a fully-Bayesian active learning framework for integrated causal discovery and reasoning. ABCI jointly infers a posterior over causal models and queries of interest. We show that our approach is more data-efficient than several baselines that only focus on learning the full causal graph.
  arXiv  Detail & Related papers  (2022-06-04T22:38:57Z)
• Causal Expectation-Maximisation [70.45873402967297]
  We show that causal inference is NP-hard even in models characterised by polytree-shaped graphs. We introduce the causal EM algorithm to reconstruct the uncertainty about the latent variables from data about categorical manifest variables. We argue that there appears to be an unnoticed limitation to the trending idea that counterfactual bounds can often be computed without knowledge of the structural equations.
  arXiv  Detail & Related papers  (2020-11-04T10:25:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.