CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists

• URL: http://arxiv.org/abs/2605.26029v2
• Date: Thu, 28 May 2026 01:38:42 GMT
• Title: CausaLab: A Scalable Environment for Interactive Causal Discovery Toward AI Scientists
• Abstract summary: We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents.<n>Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism.
• Score: 28.253879252786632
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce CausaLab, a scalable environment for evaluating interactive causal discovery by LLM agents. Unlike prior evaluations, CausaLab evaluates both whether an agent can solve a problem using causal evidence and whether its answer is grounded in a faithful recovered causal mechanism. Each episode places an agent in a synthetic laboratory: it receives prior measurement records, intervenes on a manipulator crystal, and predicts the resonance frequency of a held-out reactor crystal governed by the same mechanism. The hidden data-generating process is a randomly sampled structural causal model (SCM), so success requires recovering both a causal graph and structural equations rather than recalling prior knowledge. Experiments show a persistent gap between prediction and mechanism recovery: in the purely observational 6-node setting, GPT-5.2-high reaches 92% task accuracy but only 0.471 all-edge $F_1$. Mixed observation-intervention strategies improve structural fidelity, while pure intervention remains difficult even for strong agents. We identify premature stopping as a major weakness and show that consistency verification mitigates it. CausaLab therefore separates predictive success from causal understanding and exposes current LLM agents' limits as experimental causal reasoners.

Related papers

• Agentic Root Cause Analysis through Evidence-Grounded Reasoning [16.37572484400141]
  AgentRCA is a zero-shot agentic framework for evidence-grounded root cause analysis.<n>It performs inference-time reasoning by combining a data-driven digital twin with a tool-augmented large language model.<n>It achieves diagnostic performance competitive with fully supervised baselines without relying on fault-specific training.
  arXiv  Detail & Related papers  (2026-07-24T15:10:41Z)
• CausalGame: Benchmarking Causal Thinking of LLM Agents in Games [55.189910630332065]
  CausalGame is a benchmark for evaluating the causal thinking capabilities of AI Scientists.<n>We design 14 scenarios that incorporate selection bias, measurement error, and hidden confounders.<n>The best model reaches only 68.0% survival against analytical optima of 78-85%, and merely 5-7% of sessions receive credits on the causal-reasoning rubrics.
  arXiv  Detail & Related papers  (2026-07-05T13:08:51Z)
• One Reflection Is Not Enough: Self-Correcting Autonomous Research via Multi-Hypothesis Failure Attribution [85.65263343588026]
  We propose SAGE, a Self-correcting, Autonomous, Grounded Experimenter.<n>Its core mechanism, Multi-Hypothesis Failure Attribution (MHFA), treats recovery as a structured causal diagnosis.<n>On a 12-topic, 5-domain benchmark, SAGE increases metrics-bearing outputs from 42% to 92% over a baseline reflection.
  arXiv  Detail & Related papers  (2026-06-30T10:54:16Z)
• AI scientists produce results without reasoning scientifically [3.100302590436282]
  Large language model (LLM)-based systems are increasingly deployed to conduct scientific research autonomously.<n>Here, we evaluate LLM-based scientific agents across eight domains, spanning execution to hypothesis-driven inquiry.
  arXiv  Detail & Related papers  (2026-04-20T20:23:42Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• CausalT5K: Diagnosing and Informing Refusal for Trustworthy Causal Reasoning of Skepticism, Sycophancy, Detection-Correction, and Rung Collapse [1.4608214000864057]
  CausalT5K is a diagnostic benchmark of over 5,000 cases across 10 domains.<n>Unlike synthetic benchmarks, CausalT5K embeds causal traps in realistic narratives.<n>Preliminary experiments reveal a Four-Quadrant Control Landscape where static audit policies universally fail.
  arXiv  Detail & Related papers  (2026-02-09T17:36:56Z)
• Heterogeneous Causal Discovery of Repeated Undesirable Health Outcomes [8.16644941863291]
  Causal discovery offers an alternative to conventional approaches by generating cause-and-effect hypotheses from observational data.<n>It often relies on strong or untestable assumptions, which can limit its practical application.<n>This work aims to make causal discovery more practical by considering multiple assumptions and identifying heterogeneous effects.
  arXiv  Detail & Related papers  (2025-03-14T15:05:17Z)
• Context-Aware Reasoning On Parametric Knowledge for Inferring Causal Variables [49.31233968546582]
  We introduce a novel benchmark where the objective is to complete a partial causal graph.<n>We show the strong ability of LLMs to hypothesize the backdoor variables between a cause and its effect.<n>Unlike simple memorization of fixed associations, our task requires the LLM to reason according to the context of the entire graph.
  arXiv  Detail & Related papers  (2024-09-04T10:37:44Z)
• Smoke and Mirrors in Causal Downstream Tasks [59.90654397037007]
  This paper looks at the causal inference task of treatment effect estimation, where the outcome of interest is recorded in high-dimensional observations.<n>We compare 6 480 models fine-tuned from state-of-the-art visual backbones, and find that the sampling and modeling choices significantly affect the accuracy of the causal estimate.<n>Our results suggest that future benchmarks should carefully consider real downstream scientific questions, especially causal ones.
  arXiv  Detail & Related papers  (2024-05-27T13:26:34Z)
• Nonparametric Identifiability of Causal Representations from Unknown Interventions [63.1354734978244]
  We study causal representation learning, the task of inferring latent causal variables and their causal relations from mixtures of the variables. Our goal is to identify both the ground truth latents and their causal graph up to a set of ambiguities which we show to be irresolvable from interventional data.
  arXiv  Detail & Related papers  (2023-06-01T10:51:58Z)
• BaCaDI: Bayesian Causal Discovery with Unknown Interventions [118.93754590721173]
  BaCaDI operates in the continuous space of latent probabilistic representations of both causal structures and interventions. In experiments on synthetic causal discovery tasks and simulated gene-expression data, BaCaDI outperforms related methods in identifying causal structures and intervention targets.
  arXiv  Detail & Related papers  (2022-06-03T16:25:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.