CausalDS: Benchmarking Causal Reasoning in Data-Science Agents

• URL: http://arxiv.org/abs/2607.08093v1
• Date: Thu, 09 Jul 2026 04:03:26 GMT
• Title: CausalDS: Benchmarking Causal Reasoning in Data-Science Agents
• Abstract summary: CausalDS is a benchmark for evaluating causal reasoning in agentic data-science.<n>We ground the composition of the benchmark in empirical distributions obtained from real-world datasets.<n>The benchmark jointly evaluates symbolic causal reasoning, data science, uncertainty, abstention, and tool use/coding.
• Score: 21.447650293970458
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) increasingly act as integrated data-science agents, combining abstract reasoning with advanced tool use. Yet the relevant benchmark landscape largely divides into symbolic causal reasoning benchmarks without realistic data analysis or data analysis benchmarks without a principled causal data-generating structure. Furthermore, existing causal evaluation datasets are often restricted to curated examples from existing sources, with diversity coming from limited templatized variations rather than from systematic generation of novel synthetic causal structures. We introduce CausalDS, a benchmark for evaluating causal reasoning in agentic data-science workflows. Each benchmark instance is a scene consisting of a sampled structural causal model (SCM) with generated observational data and an accompanying synthetic natural-language story grounded in a realistic domain. We optionally ground the composition of the benchmark components in empirical distributions obtained from real-world datasets, thus retaining empirical structure while reducing the "causal parrot" risk through completely synthetic generation. From each scene, we then derive tasks spanning all three of Pearl's rungs, with typical data-science prediction tasks appearing as Rung 1. Most tasks include a data science coding component, where the model typically needs to use several tools to arrive at the final answer due to the frequent presence of imperfect observations, which are generated by an observation model. Additionally, recognizing when a question admits no warranted answer and abstaining is treated as a first-class scored outcome. The benchmark thus jointly evaluates symbolic causal reasoning, data science, uncertainty quantification, abstention, and tool use/coding.

Related papers

• Causal Ensemble Agent: Hierarchical Causal Discovery with LLM-guided Expert Reweighting [95.24017293537025]
  Causal Ensemble Agent (CEA) is a novel framework that aggregates structural insights from statistical discovery experts across different graph levels.<n>CEA achieves the strongest overall performance across a wide range of causal discovery methods.
  arXiv  Detail & Related papers  (2026-06-09T09:09:07Z)
• Leveraging Large Language Models for Causal Discovery: a Constraint-based, Argumentation-driven Approach [9.175642602891939]
  Causal Assumption-based Argumentation (ABA) is a framework that uses symbolic reasoning to ensure correspondence between input constraints and output graphs.<n>We explore the use of large language models (LLMs) as imperfect experts for Causal ABA, eliciting semantic structural priors from variable names and descriptions.<n> Experiments on standard benchmarks and semantically grounded synthetic graphs demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-18T14:15:21Z)
• CALM: A Causal Analysis Language Model for Tabular Data in Complex Systems with Local Scores, Conditional Independence Tests, and Relation Attributes [15.298086464296235]
  Causal discovery from observational data is fundamental to scientific fields like biology.<n>Existing methods, including constraint-based and score-based approaches, face significant limitations.<n>We introduce CALM, a novel causal analysis language model specifically designed for tabular data.
  arXiv  Detail & Related papers  (2025-10-10T20:19:20Z)
• Beyond Memorization: Reasoning-Driven Synthesis as a Mitigation Strategy Against Benchmark Contamination [77.69093448529455]
  We present an empirical study using an infinitely scalable framework to synthesize research-level QA directly from arXiv papers.<n>We evaluate a lack of significant performance decay near knowledge cutoff dates for models of various sizes, developers, and release dates.<n>We hypothesize that the multi-step reasoning required by our synthesis pipeline offered additional complexity that goes deeper than shallow memorization.
  arXiv  Detail & Related papers  (2025-08-26T16:41:37Z)
• Valid Inference with Imperfect Synthetic Data [39.10587411316875]
  We introduce a new estimator based on generalized method of moments.<n>We find that interactions between the moment residuals of synthetic data and those of real data can greatly improve estimates of the target parameter.
  arXiv  Detail & Related papers  (2025-08-08T18:32:52Z)
• Temporal Causal-based Simulation for Realistic Time-series Generation [1.49201581313345]
  Causal Discovery plays a pivotal role in revealing relationships among observed variables, particularly in the temporal setup.<n>Generation techniques depending on simplified assumptions on causal structure, effects and time, limit the quality and diversity of the simulated data.<n>We introduce Temporal Causal-based Simulation (TCS), a robust framework for generating realistic time-series data and their associated temporal causal graphs.
  arXiv  Detail & Related papers  (2025-06-02T10:59:48Z)
• CAnDOIT: Causal Discovery with Observational and Interventional Data from Time-Series [4.008958683836471]
  CAnDOIT is a causal discovery method to reconstruct causal models using both observational and interventional data. The use of interventional data in the causal analysis is crucial for real-world applications, such as robotics. A Python implementation of CAnDOIT has also been developed and is publicly available on GitHub.
  arXiv  Detail & Related papers  (2024-10-03T13:57:08Z)
• Discovering Mixtures of Structural Causal Models from Time Series Data [23.18511951330646]
  We propose a general variational inference-based framework called MCD to infer the underlying causal models. Our approach employs an end-to-end training process that maximizes an evidence-lower bound for the data likelihood. We demonstrate that our method surpasses state-of-the-art benchmarks in causal discovery tasks.
  arXiv  Detail & Related papers  (2023-10-10T05:13:10Z)
• Using causal inference to avoid fallouts in data-driven parametric analysis: a case study in the architecture, engineering, and construction industry [0.7566148383213173]
  The decision-making process in real-world implementations has been affected by a growing reliance on data-driven models. We investigated the synergetic pattern between the data-driven methods, empirical domain knowledge, and first-principles simulations.
  arXiv  Detail & Related papers  (2023-09-11T13:54:58Z)
• Inducing Causal Structure for Abstractive Text Summarization [76.1000380429553]
  We introduce a Structural Causal Model (SCM) to induce the underlying causal structure of the summarization data. We propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq) to learn the causal representations that can mimic the causal factors. Experimental results on two widely used text summarization datasets demonstrate the advantages of our approach.
  arXiv  Detail & Related papers  (2023-08-24T16:06:36Z)
• Amortized Inference for Causal Structure Learning [72.84105256353801]
  Learning causal structure poses a search problem that typically involves evaluating structures using a score or independence test. We train a variational inference model to predict the causal structure from observational/interventional data. Our models exhibit robust generalization capabilities under substantial distribution shift.
  arXiv  Detail & Related papers  (2022-05-25T17:37:08Z)
• A comprehensive comparative evaluation and analysis of Distributional Semantic Models [61.41800660636555]
  We perform a comprehensive evaluation of type distributional vectors, either produced by static DSMs or obtained by averaging the contextualized vectors generated by BERT. The results show that the alleged superiority of predict based models is more apparent than real, and surely not ubiquitous. We borrow from cognitive neuroscience the methodology of Representational Similarity Analysis (RSA) to inspect the semantic spaces generated by distributional models.
  arXiv  Detail & Related papers  (2021-05-20T15:18:06Z)
• CausalVAE: Structured Causal Disentanglement in Variational Autoencoder [52.139696854386976]
  The framework of variational autoencoder (VAE) is commonly used to disentangle independent factors from observations. We propose a new VAE based framework named CausalVAE, which includes a Causal Layer to transform independent factors into causal endogenous ones. Results show that the causal representations learned by CausalVAE are semantically interpretable, and their causal relationship as a Directed Acyclic Graph (DAG) is identified with good accuracy.
  arXiv  Detail & Related papers  (2020-04-18T20:09:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.