When simulations look right but causal effects go wrong: Large language models as behavioral simulators

• URL: http://arxiv.org/abs/2604.02458v1
• Date: Thu, 02 Apr 2026 18:42:29 GMT
• Title: When simulations look right but causal effects go wrong: Large language models as behavioral simulators
• Abstract summary: Large language models (LLMs) enable researchers to specify population characteristics and intervention context in natural language.<n>We evaluated three LLMs on 11 climate-psychology interventions using a dataset of 59,508 participants from 62 countries.<n> descriptive fit did not reliably translate into causal fidelity (i.e., accurate estimates of intervention effects)
• Score: 10.754758699423022
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Behavioral simulation is increasingly used to anticipate responses to interventions. Large language models (LLMs) enable researchers to specify population characteristics and intervention context in natural language, but it remains unclear to what extent LLMs can use these inputs to infer intervention effects. We evaluated three LLMs on 11 climate-psychology interventions using a dataset of 59,508 participants from 62 countries, and replicated the main analysis in two additional datasets (12 and 27 countries). LLMs reproduced observed patterns in attitudinal outcomes (e.g., climate beliefs and policy support) reasonably well, and prompting refinements improved this descriptive fit. However, descriptive fit did not reliably translate into causal fidelity (i.e., accurate estimates of intervention effects), and these two dimensions of accuracy followed different error structures. This descriptive-causal divergence held across the three datasets, but varied across intervention logics, with larger errors for interventions that depended on evoking internal experience than on directly conveying reasons or social cues. It was more pronounced for behavioral outcomes, where LLMs imposed stronger attitude-behavior coupling than in human data. Countries and population groups appearing well captured descriptively were not necessarily those with lower causal errors. Relying on descriptive fit alone may therefore create unwarranted confidence in simulation results, misleading conclusions about intervention effects and masking population disparities that matter for fairness.

Related papers

• Predicting Causal Effects from Natural Language Queries using Structured Representations [7.749911374727115]
  We introduce a new large-scale benchmark consisting of more than 72,000 natural language questions aligned with experiment descriptions.<n>We then propose a two-step framework that first generates a synthetic structured representation of a query before predicting effect size using a supervised encoder model.<n>Experiments show that finetuning plays a crucial role in improving prediction performance, with absolute error reducing by -27% up to -71% compared to prompted out-of-the-box LLMs.
  arXiv  Detail & Related papers  (2026-05-28T09:04:07Z)
• The Illusion of Intervention: Your LLM-Simulated Experiment is an Observational Study [56.649987197786096]
  Large language models (LLMs) show potential as simulators of human behavior.<n>LLMs can induce unintended shifts in latent user attributes, causing user drift.<n>We formalize the confounding or selection bias that can arise due to user drift.
  arXiv  Detail & Related papers  (2026-05-20T06:09:41Z)
• CausalGaze: Unveiling Hallucinations via Counterfactual Graph Intervention in Large Language Models [20.20916393164348]
  We introduce CausalGaze, a novel hallucination detection framework based on structural causal models (SCMs)<n>Experiments demonstrate the effectiveness of CausalGaze, especially achieving over 5.2% improvement in AUROC on the TruthfulQA dataset compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-04-13T07:09:33Z)
• Race, Ethnicity and Their Implication on Bias in Large Language Models [9.202525724606188]
  We study how race and ethnicity are represented and operationalized within large language models (LLMs)<n>We find that demographic information is distributed across internal units with substantial cross-model variation.<n> Interventions suppressing such neurons reduce bias but leave substantial residual effects.
  arXiv  Detail & Related papers  (2026-01-19T09:24:24Z)
• Can Finetuing LLMs on Small Human Samples Increase Heterogeneity, Alignment, and Belief-Action Coherence? [9.310571879281186]
  Large language models (LLMs) can serve as substitutes for human participants in survey and experimental research.<n>LLMs often fail to align with real human behavior, exhibiting limited diversity, systematic misalignment for minority subgroups, insufficient within-group variance, and discrepancies between stated beliefs and actions.<n>This study examines whether fine-tuning on a small subset of human survey data, such as that obtainable from a pilot study, can mitigate these issues and yield realistic simulated outcomes.
  arXiv  Detail & Related papers  (2025-11-26T09:50:42Z)
• HALF: Harm-Aware LLM Fairness Evaluation Aligned with Deployment [52.374772443536045]
  HALF (Harm-Aware LLM Fairness) is a framework that assesses model bias in realistic applications and weighs the outcomes by harm severity.<n>We show that HALF exposes a clear gap between previous benchmarking success and deployment readiness.
  arXiv  Detail & Related papers  (2025-10-14T07:13:26Z)
• Effectiveness of Large Language Models in Simulating Regional Psychological Structures: An Empirical Examination of Personality and Subjective Well-being [0.0]
  This study examines whether LLMs can simulate culturally grounded psychological patterns based on demographic information.<n>Simulated participants scored lower in extraversion and openness, higher in agreeableness and neuroticism, and consistently reported lower happiness.
  arXiv  Detail & Related papers  (2025-09-29T09:12:18Z)
• Can Large Language Models Help Experimental Design for Causal Discovery? [94.66802142727883]
  Large Language Model Guided Intervention Targeting (LeGIT) is a robust framework that effectively incorporates LLMs to augment existing numerical approaches for the intervention targeting in causal discovery.<n>LeGIT demonstrates significant improvements and robustness over existing methods and even surpasses humans.
  arXiv  Detail & Related papers  (2025-03-03T03:43:05Z)
• ExpliCa: Evaluating Explicit Causal Reasoning in Large Language Models [75.05436691700572]
  We introduce ExpliCa, a new dataset for evaluating Large Language Models (LLMs) in explicit causal reasoning.<n>We tested seven commercial and open-source LLMs on ExpliCa through prompting and perplexity-based metrics.<n>Surprisingly, models tend to confound temporal relations with causal ones, and their performance is also strongly influenced by the linguistic order of the events.
  arXiv  Detail & Related papers  (2025-02-21T14:23:14Z)
• LLM Targeted Underperformance Disproportionately Impacts Vulnerable Users [21.30511809806526]
  We investigate how the quality of large language models (LLMs) changes in terms of information accuracy, truthfulness, and refusals depending on user traits.<n>Our findings suggest that undesirable behaviors in state-of-the-art LLMs occur disproportionately more for users with lower English proficiency, of lower education status, and originating from outside the US.
  arXiv  Detail & Related papers  (2024-06-25T17:24:07Z)
• Evaluating Interventional Reasoning Capabilities of Large Language Models [58.52919374786108]
  Large language models (LLMs) are used to automate decision-making tasks.<n>In this paper, we evaluate whether LLMs can accurately update their knowledge of a data-generating process in response to an intervention.<n>We create benchmarks that span diverse causal graphs (e.g., confounding, mediation) and variable types.<n>These benchmarks allow us to isolate the ability of LLMs to accurately predict changes resulting from their ability to memorize facts or find other shortcuts.
  arXiv  Detail & Related papers  (2024-04-08T14:15:56Z)
• Efficient Causal Inference from Combined Observational and Interventional Data through Causal Reductions [68.6505592770171]
  Unobserved confounding is one of the main challenges when estimating causal effects. We propose a novel causal reduction method that replaces an arbitrary number of possibly high-dimensional latent confounders. We propose a learning algorithm to estimate the parameterized reduced model jointly from observational and interventional data.
  arXiv  Detail & Related papers  (2021-03-08T14:29:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.