Testing Generalizability in Causal Inference

• URL: http://arxiv.org/abs/2411.03021v2
• Date: Thu, 12 Jun 2025 14:02:34 GMT
• Title: Testing Generalizability in Causal Inference
• Authors: Daniel de Vassimon Manela, Linying Yang, Robin J. Evans,
• Abstract summary: No formal procedure exists for statistically evaluating generalizability in machine learning algorithms.<n>We propose a systematic framework for statistically evaluating the generalizability of high-dimensional causal inference models.
• Score: 3.547529079746247
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ensuring robust model performance in diverse real-world scenarios requires addressing generalizability across domains with covariate shifts. However, no formal procedure exists for statistically evaluating generalizability in machine learning algorithms. Existing predictive metrics like mean squared error (MSE) help to quantify the relative performance between models, but do not directly answer whether a model can or cannot generalize. To address this gap in the domain of causal inference, we propose a systematic framework for statistically evaluating the generalizability of high-dimensional causal inference models. Our approach uses the frugal parameterization to flexibly simulate from fully and semi-synthetic causal benchmarks, offering a comprehensive evaluation for both mean and distributional regression methods. Grounded in real-world data, our method ensures more realistic evaluations, which is often missing in current work relying on simplified datasets. Furthermore, using simulations and statistical testing, our framework is robust and avoids over-reliance on conventional metrics, providing statistical safeguards for decision making.

Related papers

• Frugal, Flexible, Faithful: Causal Data Simulation via Frengression [4.446798246007668]
  We introduce frengression, a deep generative realization of the frugal parameterization.<n>frengression provides accurate estimation and flexible, faithful simulation of time-varying data.<n>We envision this framework sparking new research into generative approaches for causal margin modelling.
  arXiv  Detail & Related papers  (2025-08-01T18:43:59Z)
• Partial Transportability for Domain Generalization [56.37032680901525]
  Building on the theory of partial identification and transportability, this paper introduces new results for bounding the value of a functional of the target distribution.<n>Our contribution is to provide the first general estimation technique for transportability problems.<n>We propose a gradient-based optimization scheme for making scalable inferences in practice.
  arXiv  Detail & Related papers  (2025-03-30T22:06:37Z)
• Simulation-based Bayesian inference under model misspecification [4.2490325931915285]
  We focus on the challenges faced by SBI methods under model misspecification. We consolidate recent research aimed at mitigating the effects of misspecification. To illustrate both the vulnerabilities of popular SBI methods and the effectiveness of misspecification-robust alternatives, we present empirical results.
  arXiv  Detail & Related papers  (2025-03-16T01:47:19Z)
• Meta-Statistical Learning: Supervised Learning of Statistical Inference [59.463430294611626]
  This work demonstrates that the tools and principles driving the success of large language models (LLMs) can be repurposed to tackle distribution-level tasks. We propose meta-statistical learning, a framework inspired by multi-instance learning that reformulates statistical inference tasks as supervised learning problems.
  arXiv  Detail & Related papers  (2025-02-17T18:04:39Z)
• Model-free Methods for Event History Analysis and Efficient Adjustment (PhD Thesis) [55.2480439325792]
  This thesis is a series of independent contributions to statistics unified by a model-free perspective.<n>The first chapter elaborates on how a model-free perspective can be used to formulate flexible methods that leverage prediction techniques from machine learning.<n>The second chapter studies the concept of local independence, which describes whether the evolution of one process is directly influenced by another.
  arXiv  Detail & Related papers  (2025-02-11T19:24:09Z)
• On conditional diffusion models for PDE simulations [53.01911265639582]
  We study score-based diffusion models for forecasting and assimilation of sparse observations. We propose an autoregressive sampling approach that significantly improves performance in forecasting. We also propose a new training strategy for conditional score-based models that achieves stable performance over a range of history lengths.
  arXiv  Detail & Related papers  (2024-10-21T18:31:04Z)
• MITA: Bridging the Gap between Model and Data for Test-time Adaptation [68.62509948690698]
  Test-Time Adaptation (TTA) has emerged as a promising paradigm for enhancing the generalizability of models. We propose Meet-In-The-Middle based MITA, which introduces energy-based optimization to encourage mutual adaptation of the model and data from opposing directions.
  arXiv  Detail & Related papers  (2024-10-12T07:02:33Z)
• Addressing Misspecification in Simulation-based Inference through Data-driven Calibration [43.811367860375825]
  Recent work has demonstrated that model misspecification can harm simulation-based inference's reliability. This work introduces robust posterior estimation (ROPE), a framework that overcomes model misspecification with a small real-world calibration set of ground truth parameter measurements.
  arXiv  Detail & Related papers  (2024-05-14T16:04:39Z)
• Bayesian Nonparametrics Meets Data-Driven Distributionally Robust Optimization [29.24821214671497]
  Training machine learning and statistical models often involve optimizing a data-driven risk criterion. We propose a novel robust criterion by combining insights from Bayesian nonparametric (i.e., Dirichlet process) theory and a recent decision-theoretic model of smooth ambiguity-averse preferences. For practical implementation, we propose and study tractable approximations of the criterion based on well-known Dirichlet process representations.
  arXiv  Detail & Related papers  (2024-01-28T21:19:15Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Learning Robust Statistics for Simulation-based Inference under Model Misspecification [23.331522354991527]
  We propose the first general approach to handle model misspecification that works across different classes of simulation-based inference methods. We show that our method yields robust inference in misspecified scenarios, whilst still being accurate when the model is well-specified.
  arXiv  Detail & Related papers  (2023-05-25T09:06:26Z)
• Variable Importance Matching for Causal Inference [73.25504313552516]
  We describe a general framework called Model-to-Match that achieves these goals. Model-to-Match uses variable importance measurements to construct a distance metric. We operationalize the Model-to-Match framework with LASSO.
  arXiv  Detail & Related papers  (2023-02-23T00:43:03Z)
• One-Shot Domain Adaptive and Generalizable Semantic Segmentation with Class-Aware Cross-Domain Transformers [96.51828911883456]
  Unsupervised sim-to-real domain adaptation (UDA) for semantic segmentation aims to improve the real-world test performance of a model trained on simulated data. Traditional UDA often assumes that there are abundant unlabeled real-world data samples available during training for the adaptation. We explore the one-shot unsupervised sim-to-real domain adaptation (OSUDA) and generalization problem, where only one real-world data sample is available.
  arXiv  Detail & Related papers  (2022-12-14T15:54:15Z)
• Rigorous Assessment of Model Inference Accuracy using Language Cardinality [5.584832154027001]
  We develop a systematic approach that minimizes bias and uncertainty in model accuracy assessment by replacing statistical estimation with deterministic accuracy measures. We experimentally demonstrate the consistency and applicability of our approach by assessing the accuracy of models inferred by state-of-the-art inference tools.
  arXiv  Detail & Related papers  (2022-11-29T21:03:26Z)
• CausalBench: A Large-scale Benchmark for Network Inference from Single-cell Perturbation Data [61.088705993848606]
  We introduce CausalBench, a benchmark suite for evaluating causal inference methods on real-world interventional data. CaulBench incorporates biologically-motivated performance metrics, including new distribution-based interventional metrics.
  arXiv  Detail & Related papers  (2022-10-31T13:04:07Z)
• HyperImpute: Generalized Iterative Imputation with Automatic Model Selection [77.86861638371926]
  We propose a generalized iterative imputation framework for adaptively and automatically configuring column-wise models. We provide a concrete implementation with out-of-the-box learners, simulators, and interfaces.
  arXiv  Detail & Related papers  (2022-06-15T19:10:35Z)
• Variational Gibbs Inference for Statistical Model Estimation from Incomplete Data [7.4250022679087495]
  We introduce variational Gibbs inference (VGI), a new general-purpose method to estimate the parameters of statistical models from incomplete data. We validate VGI on a set of synthetic and real-world estimation tasks, estimating important machine learning models such as variational autoencoders and normalising flows from incomplete data.
  arXiv  Detail & Related papers  (2021-11-25T17:22:22Z)
• MINIMALIST: Mutual INformatIon Maximization for Amortized Likelihood Inference from Sampled Trajectories [61.3299263929289]
  Simulation-based inference enables learning the parameters of a model even when its likelihood cannot be computed in practice. One class of methods uses data simulated with different parameters to infer an amortized estimator for the likelihood-to-evidence ratio. We show that this approach can be formulated in terms of mutual information between model parameters and simulated data.
  arXiv  Detail & Related papers  (2021-06-03T12:59:16Z)
• Distributed Learning of Finite Gaussian Mixtures [21.652015112462]
  We study split-and-conquer approaches for the distributed learning of finite Gaussian mixtures. New estimator is shown to be consistent and retains root-n consistency under some general conditions. Experiments based on simulated and real-world data show that the proposed split-and-conquer approach has comparable statistical performance with the global estimator.
  arXiv  Detail & Related papers  (2020-10-20T16:17:47Z)
• Model-based Policy Optimization with Unsupervised Model Adaptation [37.09948645461043]
  We investigate how to bridge the gap between real and simulated data due to inaccurate model estimation for better policy optimization. We propose a novel model-based reinforcement learning framework AMPO, which introduces unsupervised model adaptation. Our approach achieves state-of-the-art performance in terms of sample efficiency on a range of continuous control benchmark tasks.
  arXiv  Detail & Related papers  (2020-10-19T14:19:42Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)
• Amortized Bayesian model comparison with evidential deep learning [0.12314765641075436]
  We propose a novel method for performing Bayesian model comparison using specialized deep learning architectures. Our method is purely simulation-based and circumvents the step of explicitly fitting all alternative models under consideration to each observed dataset. We show that our method achieves excellent results in terms of accuracy, calibration, and efficiency across the examples considered in this work.
  arXiv  Detail & Related papers  (2020-04-22T15:15:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.