CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation

• URL: http://arxiv.org/abs/2602.20571v1
• Date: Tue, 24 Feb 2026 05:44:25 GMT
• Title: CausalReasoningBenchmark: A Real-World Benchmark for Disentangled Evaluation of Causal Identification and Estimation
• Abstract summary: CausalReasoningBenchmark is a benchmark of 173 queries across 138 real-world datasets.<n>It distinguishes failures in causal reasoning from errors in numerical execution.
• Score: 18.187891198793455
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Many benchmarks for automated causal inference evaluate a system's performance based on a single numerical output, such as an Average Treatment Effect (ATE). This approach conflates two distinct steps in causal analysis: identification-formulating a valid research design under stated assumptions-and estimation-implementing that design numerically on finite data. We introduce CausalReasoningBenchmark, a benchmark of 173 queries across 138 real-world datasets, curated from 85 peer-reviewed research papers and four widely-used causal-inference textbooks. For each query a system must produce (i) a structured identification specification that names the strategy, the treatment, outcome, and control variables, and all design-specific elements, and (ii) a point estimate with a standard error. By scoring these two components separately, our benchmark enables granular diagnosis: it distinguishes failures in causal reasoning from errors in numerical execution. Baseline results with a state-of-the-art LLM show that, while the model correctly identifies the high-level strategy in 84 % of cases, full identification-specification correctness drops to only 30 %, revealing that the bottleneck lies in the nuanced details of research design rather than in computation. CausalReasoningBenchmark is publicly available on Hugging Face and is designed to foster the development of more robust automated causal-inference systems.

Related papers

• Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibility [7.838210213616623]
  We develop a systematic account of test-time scaling along three axes.<n>First, we formalize test-time scaling as budgeted inference over the implicit prefix tree of an autoregressive model.<n>Second, we treat the evaluated object as the entire inference system.
  arXiv  Detail & Related papers  (2026-08-04T17:57:20Z)
• Towards Diverse and Comprehensive Benchmarks for Mutual Information Estimation [47.29900218955283]
  Mutual information estimation is a central problem in machine learning and statistics.<n>Existing benchmarks evaluate estimators on simplified, low-dimensional distributions, leaving their performance on complex, realistic data largely unexplored.<n>We address this gap with a comprehensive benchmarking framework grounded in a unified copula-theoretic perspective.
  arXiv  Detail & Related papers  (2026-07-03T16:55:55Z)
• Causal Ensemble Agent: Hierarchical Causal Discovery with LLM-guided Expert Reweighting [95.24017293537025]
  Causal Ensemble Agent (CEA) is a novel framework that aggregates structural insights from statistical discovery experts across different graph levels.<n>CEA achieves the strongest overall performance across a wide range of causal discovery methods.
  arXiv  Detail & Related papers  (2026-06-09T09:09:07Z)
• Computational Identifiability [58.12019389497595]
  'Identical identifiability' assumes properties, infinite data, or other mathematically idealized conditions.<n>'Computational identifiability' is to instead define a finite computational search procedure for an empirical estimator.<n>We show how this framework allows us to answer fine-grained, practical identification questions.
  arXiv  Detail & Related papers  (2026-06-08T19:39:59Z)
• Learning More from Less: Unlocking Internal Representations for Benchmark Compression [37.69575776639016]
  We introduce REPCORE, which aligns heterogeneous hidden states into a unified latent space to construct representative coresets.<n>Experiments on five benchmarks and over 200 models show consistent gains over output-based baselines in ranking correlation and estimation accuracy.
  arXiv  Detail & Related papers  (2026-01-31T13:11:39Z)
• Benchmark^2: Systematic Evaluation of LLM Benchmarks [66.2731798872668]
  We propose Benchmark2, a comprehensive framework comprising three complementary metrics.<n>We conduct experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains.<n>Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction can achieve comparable evaluation performance.
  arXiv  Detail & Related papers  (2026-01-07T14:59:03Z)
• Fantastic Bugs and Where to Find Them in AI Benchmarks [28.604919035475188]
  We introduce a framework for systematic benchmark revision that leverages statistical analysis of response patterns to flag potentially invalid questions.<n>Our approach builds on a core assumption commonly used in AI evaluations that the mean score sufficiently summarizes model performance.<n>Across nine widely used benchmarks, our method guides expert review to identify problematic questions with up to 84% precision.
  arXiv  Detail & Related papers  (2025-11-20T22:49:21Z)
• Score-based Greedy Search for Structure Identification of Partially Observed Linear Causal Models [34.09555821357439]
  We propose the first score-based greedy search method for the identification of structure involving latent variables with identifiability guarantees.<n>We then design Latent variable Greedy Equivalence Search (LGES), a greedy search algorithm for this class of model with well-defined operators.
  arXiv  Detail & Related papers  (2025-10-05T21:50:17Z)
• Garbage In, Reasoning Out? Why Benchmark Scores are Unreliable and What to Do About It [1.6261897792391753]
  We conduct a systematic audit of three widely used reasoning benchmarks, SocialIQa, FauxPas-EAI, and ToMi.<n>We uncover pervasive flaws in both benchmark items and evaluation methodology.
  arXiv  Detail & Related papers  (2025-06-30T13:57:28Z)
• AutoLogi: Automated Generation of Logic Puzzles for Evaluating Reasoning Abilities of Large Language Models [86.83875864328984]
  We propose an automated method for synthesizing open-ended logic puzzles, and use it to develop a bilingual benchmark, AutoLogi.<n>Our approach features program-based verification and controllable difficulty levels, enabling more reliable evaluation that better distinguishes models' reasoning abilities.
  arXiv  Detail & Related papers  (2025-02-24T07:02:31Z)
• Beyond the Singular: The Essential Role of Multiple Generations in Effective Benchmark Evaluation and Analysis [10.133537818749291]
  Large language models (LLMs) have demonstrated significant utilities in real-world applications.<n> Benchmark evaluations are crucial for assessing the capabilities of LLMs.
  arXiv  Detail & Related papers  (2025-02-13T03:43:33Z)
• Language Models as Causal Effect Generators [48.696932388555894]
  We present sequence-driven structural causal models (SD-SCMs)<n>An SD-SCM enables sampling from observational, interventional, and counterfactual distributions according to the desired causal structure.<n>We propose a new type of benchmark for causal inference methods, generating individual-level counterfactual data to test treatment effect estimation.
  arXiv  Detail & Related papers  (2024-11-12T18:50:35Z)
• Evaluating Human Alignment and Model Faithfulness of LLM Rationale [66.75309523854476]
  We study how well large language models (LLMs) explain their generations through rationales. We show that prompting-based methods are less "faithful" than attribution-based explanations.
  arXiv  Detail & Related papers  (2024-06-28T20:06:30Z)
• Evaluating Generative Language Models in Information Extraction as Subjective Question Correction [49.729908337372436]
  We propose a new evaluation method, SQC-Score. Inspired by the principles in subjective question correction, we propose a new evaluation method, SQC-Score. Results on three information extraction tasks show that SQC-Score is more preferred by human annotators than the baseline metrics.
  arXiv  Detail & Related papers  (2024-04-04T15:36:53Z)
• Are LLMs Capable of Data-based Statistical and Causal Reasoning? Benchmarking Advanced Quantitative Reasoning with Data [89.2410799619405]
  We introduce the Quantitative Reasoning with Data benchmark to evaluate Large Language Models' capability in statistical and causal reasoning with real-world data. The benchmark comprises a dataset of 411 questions accompanied by data sheets from textbooks, online learning materials, and academic papers. To compare models' quantitative reasoning abilities on data and text, we enrich the benchmark with an auxiliary set of 290 text-only questions, namely QRText.
  arXiv  Detail & Related papers  (2024-02-27T16:15:03Z)
• Discover, Explanation, Improvement: An Automatic Slice Detection Framework for Natural Language Processing [72.14557106085284]
  slice detection models (SDM) automatically identify underperforming groups of datapoints. This paper proposes a benchmark named "Discover, Explain, improve (DEIM)" for classification NLP tasks. Our evaluation shows that Edisa can accurately select error-prone datapoints with informative semantic features.
  arXiv  Detail & Related papers  (2022-11-08T19:00:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.