Predicting Causal Effects from Natural Language Queries using Structured Representations

• URL: http://arxiv.org/abs/2605.29631v1
• Date: Thu, 28 May 2026 09:04:07 GMT
• Title: Predicting Causal Effects from Natural Language Queries using Structured Representations
• Abstract summary: We introduce a new large-scale benchmark consisting of more than 72,000 natural language questions aligned with experiment descriptions.<n>We then propose a two-step framework that first generates a synthetic structured representation of a query before predicting effect size using a supervised encoder model.<n>Experiments show that finetuning plays a crucial role in improving prediction performance, with absolute error reducing by -27% up to -71% compared to prompted out-of-the-box LLMs.
• Score: 7.749911374727115
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Randomized controlled trials are a cornerstone of medicine and the social sciences as they enable reliable estimates of causal effects. However, they are costly and time-consuming to conduct, motivating interest in predicting causal effects from existing experimental evidence. Recent advances in large language models (LLMs) have demonstrated strong performance on knowledge-intensive tasks, raising the question of whether these models can be used for forecasting causal effect sizes. To investigate this, we introduce Query2Effect, a new large-scale benchmark consisting of more than 72,000 natural language questions aligned with experiment descriptions, created to simulate realistic information-seeking scenarios by varying query specificity along dimensions of implicitness, abstraction, and ambiguity. We then propose a two-step framework that first generates a synthetic structured representation of a query before predicting effect size using a supervised encoder model. Experiments show that finetuning plays a crucial role in improving prediction performance, with absolute error reducing by -27% up to -71% compared to prompted out-of-the-box LLMs, and that our two-step framework is beneficial for out-of-domain generalization, highlighting the benefits of separating semantic interpretation from numerical effect estimation.

Related papers

• Bounding the Causal Impact of ML-assisted Decision-Making via Counterfactual Correctness [2.9267249513932008]
  We present a partial-identification approach to constructing bounds on the causal effect of a new model.<n>The core innovation in our approach is to leverage assumptions relating fine-grained predictive accuracy to downstream outcomes.
  arXiv  Detail & Related papers  (2026-07-23T20:45:10Z)
• Beyond Inefficiency: Systemic Costs of Incivility in Multi-Agent Monte Carlo Simulations [0.0]
  We use Large Language Model (LLM) based Multi-Agent Systems as a controlled sociological sandbox.<n>We generate thousands of structured 1-on-1 adversarial debates across varying toxicity conditions.<n>We identify a significant first-mover advantage, whereby the agent initiating the discussion wins significantly above chance regardless of toxicity condition.
  arXiv  Detail & Related papers  (2026-05-12T08:54:03Z)
• ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Models [102.4511331368587]
  ARISE (Adaptive Resolution-aware Scaling Evaluation) is a novel metric designed to assess the test-time scaling effectiveness of large reasoning models.<n>We conduct comprehensive experiments evaluating state-of-the-art reasoning models across diverse domains.
  arXiv  Detail & Related papers  (2025-10-07T15:10:51Z)
• Do-PFN: In-Context Learning for Causal Effect Estimation [75.62771416172109]
  We show that Prior-data fitted networks (PFNs) can be pre-trained on synthetic data to predict outcomes.<n>Our approach allows for the accurate estimation of causal effects without knowledge of the underlying causal graph.
  arXiv  Detail & Related papers  (2025-06-06T12:43:57Z)
• Data Fusion for Partial Identification of Causal Effects [62.56890808004615]
  We propose a novel partial identification framework that enables researchers to answer key questions.<n>Is the causal effect positive or negative? and How severe must assumption violations be to overturn this conclusion?<n>We apply our framework to the Project STAR study, which investigates the effect of classroom size on students' third-grade standardized test performance.
  arXiv  Detail & Related papers  (2025-05-30T07:13:01Z)
• Kernel-based estimators for functional causal effects [1.6749379740049928]
  We propose causal effect estimators based on empirical Fr'echet means and operator-valued kernels, tailored to functional data spaces.<n>These methods address the challenges of high-dimensionality, sequential ordering, and model complexity while preserving robustness to treatment misspecification.
  arXiv  Detail & Related papers  (2025-03-06T22:48:55Z)
• Prediction-Powered Causal Inferences [59.98498488132307]
  We focus on Prediction-Powered Causal Inferences (PPCI)<n>We first show that conditional calibration guarantees valid PPCI at population level.<n>We then introduce a sufficient representation constraint transferring validity across experiments.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• Causal Post-Processing of Predictive Models [0.7349727826230863]
  We propose causal post-processing, a family of techniques that uses limited experimental data to refine the outputs of predictive models.<n>We show that CPP can improve intervention decisions, particularly when predictive models capture a useful but imperfect causal signal.
  arXiv  Detail & Related papers  (2024-06-13T20:18:16Z)
• Estimating the Causal Effects of Natural Logic Features in Transformer-Based NLI Models [16.328341121232484]
  We apply causal effect estimation strategies to measure the effect of context interventions. We investigate robustness to irrelevant changes and sensitivity to impactful changes of Transformers.
  arXiv  Detail & Related papers  (2024-04-03T10:22:35Z)
• A Second Look at the Impact of Passive Voice Requirements on Domain Modeling: Bayesian Reanalysis of an Experiment [4.649794383775257]
  We reanalyze the only known controlled experiment investigating the impact of passive voice on the subsequent activity of domain modeling. Our results reveal that the effects observed by the original authors turned out to be much less significant than previously assumed.
  arXiv  Detail & Related papers  (2024-02-16T16:24:00Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Generative Counterfactuals for Neural Networks via Attribute-Informed Perturbation [51.29486247405601]
  We design a framework to generate counterfactuals for raw data instances with the proposed Attribute-Informed Perturbation (AIP) By utilizing generative models conditioned with different attributes, counterfactuals with desired labels can be obtained effectively and efficiently. Experimental results on real-world texts and images demonstrate the effectiveness, sample quality as well as efficiency of our designed framework.
  arXiv  Detail & Related papers  (2021-01-18T08:37:13Z)
• Double Robust Representation Learning for Counterfactual Prediction [68.78210173955001]
  We propose a novel scalable method to learn double-robust representations for counterfactual predictions. We make robust and efficient counterfactual predictions for both individual and average treatment effects. The algorithm shows competitive performance with the state-of-the-art on real world and synthetic data.
  arXiv  Detail & Related papers  (2020-10-15T16:39:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.