End-To-End Causal Effect Estimation from Unstructured Natural Language Data

• URL: http://arxiv.org/abs/2407.07018v3
• Date: Mon, 28 Oct 2024 16:03:20 GMT
• Title: End-To-End Causal Effect Estimation from Unstructured Natural Language Data
• Authors: Nikita Dhawan, Leonardo Cotta, Karen Ullrich, Rahul G. Krishnan, Chris J. Maddison,
• Abstract summary: We show how large, diverse observational text data can be mined with large language models (LLMs) to produce inexpensive causal effect estimates. We introduce NATURAL, a novel family of causal effect estimators built with LLMs that operate over datasets of unstructured text. Our results suggest that unstructured text data is a rich source of causal effect information, and NATURAL is a first step towards an automated pipeline to tap this resource.
• Score: 23.484226791467478
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowing the effect of an intervention is critical for human decision-making, but current approaches for causal effect estimation rely on manual data collection and structuring, regardless of the causal assumptions. This increases both the cost and time-to-completion for studies. We show how large, diverse observational text data can be mined with large language models (LLMs) to produce inexpensive causal effect estimates under appropriate causal assumptions. We introduce NATURAL, a novel family of causal effect estimators built with LLMs that operate over datasets of unstructured text. Our estimators use LLM conditional distributions (over variables of interest, given the text data) to assist in the computation of classical estimators of causal effect. We overcome a number of technical challenges to realize this idea, such as automating data curation and using LLMs to impute missing information. We prepare six (two synthetic and four real) observational datasets, paired with corresponding ground truth in the form of randomized trials, which we used to systematically evaluate each step of our pipeline. NATURAL estimators demonstrate remarkable performance, yielding causal effect estimates that fall within 3 percentage points of their ground truth counterparts, including on real-world Phase 3/4 clinical trials. Our results suggest that unstructured text data is a rich source of causal effect information, and NATURAL is a first step towards an automated pipeline to tap this resource.

Related papers

• Causal Effect Estimation with Latent Textual Treatments [9.451877252547197]
  We present an end-to-end pipeline for the generation and causal estimation of latent textual interventions.<n>Our work first performs hypothesis generation and steering via sparse autoencoders (SAEs), followed by robust causal estimation.
  arXiv  Detail & Related papers  (2026-02-17T17:06:12Z)
• Information Capacity: Evaluating the Efficiency of Large Language Models via Text Compression [53.39128997308138]
  We introduce information capacity, a measure of model efficiency based on text compression performance.<n> Empirical evaluations on mainstream open-source models show that models of varying sizes within a series exhibit consistent information capacity.<n>A distinctive feature of information capacity is that it incorporates tokenizer efficiency, which affects both input and output token counts.
  arXiv  Detail & Related papers  (2025-11-11T10:07:32Z)
• CausalPFN: Amortized Causal Effect Estimation via In-Context Learning [15.645599403885605]
  CausalPFN infers causal effects for new observational datasets out-of-the-box.<n>Our approach achieves superior average performance on heterogeneous and average treatment effect estimation benchmarks.<n>CausalPFN provides calibrated uncertainty estimates to support reliable decision-making based on Bayesian principles.
  arXiv  Detail & Related papers  (2025-06-09T16:31:06Z)
• Do-PFN: In-Context Learning for Causal Effect Estimation [75.62771416172109]
  We show that Prior-data fitted networks (PFNs) can be pre-trained on synthetic data to predict outcomes.<n>Our approach allows for the accurate estimation of causal effects without knowledge of the underlying causal graph.
  arXiv  Detail & Related papers  (2025-06-06T12:43:57Z)
• Data Fusion for Partial Identification of Causal Effects [62.56890808004615]
  We propose a novel partial identification framework that enables researchers to answer key questions.<n>Is the causal effect positive or negative? and How severe must assumption violations be to overturn this conclusion?<n>We apply our framework to the Project STAR study, which investigates the effect of classroom size on students' third-grade standardized test performance.
  arXiv  Detail & Related papers  (2025-05-30T07:13:01Z)
• Self-Rationalization in the Wild: A Large Scale Out-of-Distribution Evaluation on NLI-related tasks [59.47851630504264]
  Free-text explanations are expressive and easy to understand, but many datasets lack annotated explanation data. We fine-tune T5-Large and OLMo-7B models and assess the impact of fine-tuning data quality, the number of fine-tuning samples, and few-shot selection methods. The models are evaluated on 19 diverse OOD datasets across three tasks: natural language inference (NLI), fact-checking, and hallucination detection in abstractive summarization.
  arXiv  Detail & Related papers  (2025-02-07T10:01:32Z)
• Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning [61.99353167168545]
  We show that fine-tuning with LLM-generated data improves target task performance and reduces non-target task degradation.<n>This is the first work to provide an empirical explanation based on token perplexity reduction to mitigate catastrophic forgetting in LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2025-01-24T08:18:56Z)
• Marginal Causal Flows for Validation and Inference [3.547529079746247]
  Investigating the marginal causal effect of an intervention on an outcome from complex data remains challenging. We introduce Frugal Flows, a novel likelihood-based machine learning model that uses normalising flows to flexibly learn the data-generating process. We demonstrate the above with experiments on both simulated and real-world datasets.
  arXiv  Detail & Related papers  (2024-11-02T16:04:57Z)
• Entropy Law: The Story Behind Data Compression and LLM Performance [115.70395740286422]
  We find that model performance is negatively correlated to the compression ratio of training data, which usually yields a lower training loss. Based on the findings of the entropy law, we propose a quite efficient and universal data selection method. We also present an interesting application of entropy law that can detect potential performance risks at the beginning of model training.
  arXiv  Detail & Related papers  (2024-07-09T08:14:29Z)
• Do Finetti: On Causal Effects for Exchangeable Data [45.96632286841583]
  We study causal effect estimation in a setting where the data are not i.i.d. We focus on exchangeable data satisfying an assumption of independent causal mechanisms.
  arXiv  Detail & Related papers  (2024-05-29T07:31:18Z)
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
  Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
  arXiv  Detail & Related papers  (2023-05-22T17:02:15Z)
• A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicity [84.6421260559093]
  This study is the largest set of experiments to validate, quantify, and expose undocumented intuitions about text pretraining. Our findings indicate there does not exist a one-size-fits-all solution to filtering training data.
  arXiv  Detail & Related papers  (2023-05-22T15:57:53Z)
• Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions [59.284907093349425]
  Large amounts of training data are one of the major reasons for the high performance of state-of-the-art NLP models. We provide a language for describing how training data influences predictions, through a causal framework. Our framework bypasses the need to retrain expensive models and allows us to estimate causal effects based on observational data alone.
  arXiv  Detail & Related papers  (2022-07-28T17:36:24Z)
• Improving Data-driven Heterogeneous Treatment Effect Estimation Under Structure Uncertainty [13.452510519858995]
  Estimating how a treatment affects units individually, known as heterogeneous treatment effect (HTE) estimation, is an essential part of decision-making and policy implementation. We develop a feature selection method that considers each feature's value for HTE estimation and learns the relevant parts of the causal structure from data.
  arXiv  Detail & Related papers  (2022-06-25T16:26:35Z)
• Causal Effect Estimation using Variational Information Bottleneck [19.6760527269791]
  Causal inference is to estimate the causal effect in a causal relationship when intervention is applied. We propose a method to estimate Causal Effect by using Variational Information Bottleneck (CEVIB)
  arXiv  Detail & Related papers  (2021-10-26T13:46:12Z)
• Multi-Source Causal Inference Using Control Variates [81.57072928775509]
  We propose a general algorithm to estimate causal effects from emphmultiple data sources. We show theoretically that this reduces the variance of the ATE estimate. We apply this framework to inference from observational data under an outcome selection bias.
  arXiv  Detail & Related papers  (2021-03-30T21:20:51Z)
• Provably Efficient Causal Reinforcement Learning with Confounded Observational Data [135.64775986546505]
  We study how to incorporate the dataset (observational data) collected offline, which is often abundantly available in practice, to improve the sample efficiency in the online setting. We propose the deconfounded optimistic value iteration (DOVI) algorithm, which incorporates the confounded observational data in a provably efficient manner.
  arXiv  Detail & Related papers  (2020-06-22T14:49:33Z)
• Towards unique and unbiased causal effect estimation from data with hidden variables [0.0]
  Causal effect estimation from observational data is a crucial but challenging task. We propose an approach to achieving unique and unbiased estimation of causal effects from data with hidden variables. Based on the theorems, two algorithms are proposed for finding the proper adjustment sets from data with hidden variables.
  arXiv  Detail & Related papers  (2020-02-24T06:42:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.