Efficient Randomized Experiments Using Foundation Models

• URL: http://arxiv.org/abs/2502.04262v1
• Date: Thu, 06 Feb 2025 17:54:10 GMT
• Title: Efficient Randomized Experiments Using Foundation Models
• Authors: Piersilvio De Bartolomeis, Javier Abad, Guanbo Wang, Konstantin Donhauser, Raymond M. Duch, Fanny Yang, Issa J. Dahabreh,
• Abstract summary: In this paper, we propose a novel approach that integrates the predictions from multiple foundation models while preserving valid statistical inference. Our estimator offers substantial precision gains, equivalent to a reduction of up to 20% in the sample size needed to match the same precision as the standard estimator based on experimental data alone.
• Score: 10.606998433337894
• License:
• Abstract: Randomized experiments are the preferred approach for evaluating the effects of interventions, but they are costly and often yield estimates with substantial uncertainty. On the other hand, in silico experiments leveraging foundation models offer a cost-effective alternative that can potentially attain higher statistical precision. However, the benefits of in silico experiments come with a significant risk: statistical inferences are not valid if the models fail to accurately predict experimental responses to interventions. In this paper, we propose a novel approach that integrates the predictions from multiple foundation models with experimental data while preserving valid statistical inference. Our estimator is consistent and asymptotically normal, with asymptotic variance no larger than the standard estimator based on experimental data alone. Importantly, these statistical properties hold even when model predictions are arbitrarily biased. Empirical results across several randomized experiments show that our estimator offers substantial precision gains, equivalent to a reduction of up to 20% in the sample size needed to match the same precision as the standard estimator based on experimental data alone.

Related papers

• Causal Lifting of Neural Representations: Zero-Shot Generalization for Causal Inferences [56.23412698865433]
  We focus on causal inferences on a target experiment with unlabeled factual outcomes, retrieved by a predictive model fine-tuned on a labeled similar experiment. First, we show that factual outcome estimation via Empirical Risk Minimization (ERM) may fail to yield valid causal inferences on the target population. We propose Deconfounded Empirical Risk Minimization (DERM), a new simple learning procedure minimizing the risk over a fictitious target population.
  arXiv  Detail & Related papers  (2025-02-10T10:52:17Z)
• Prediction-Guided Active Experiments [18.494123886098215]
  We introduce a new framework for active experimentation, the Prediction-Guided Active Experiment (PGAE) PGAE leverages predictions from an existing machine learning model to guide sampling and experimentation. We show that PGAE remains efficient and attains the same semi-parametric bound under certain regularity assumptions.
  arXiv  Detail & Related papers  (2024-11-18T20:16:24Z)
• Evidential time-to-event prediction with calibrated uncertainty quantification [12.446406577462069]
  Time-to-event analysis provides insights into clinical prognosis and treatment recommendations. We propose an evidential regression model specifically designed for time-to-event prediction. We show that our model delivers both accurate and reliable performance, outperforming state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-11-12T15:06:04Z)
• Statistical Inference in Tensor Completion: Optimal Uncertainty Quantification and Statistical-to-Computational Gaps [7.174572371800217]
  This paper presents a simple yet efficient method for statistical inference of tensor linear forms using incomplete and noisy observations. It is suitable for various statistical inference tasks, including constructing confidence intervals, inference under heteroskedastic and sub-exponential noise, and simultaneous testing.
  arXiv  Detail & Related papers  (2024-10-15T03:09:52Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Expected Validation Performance and Estimation of a Random Variable's Maximum [48.83713377993604]
  We analyze three statistical estimators for expected validation performance. We find the unbiased estimator has the highest variance, and the estimator with the smallest variance has the largest bias. We find that the two biased estimators lead to the fewest incorrect conclusions.
  arXiv  Detail & Related papers  (2021-10-01T18:48:47Z)
• What can the millions of random treatments in nonexperimental data reveal about causes? [0.0]
  The article introduces one such model and a Bayesian approach to combine the $O(n2)$ pairwise observations typically available in nonexperimnetal data. We demonstrate that the proposed approach recovers causal effects in common NSW samples, as well as in arbitrary subpopulations and an order-of-magnitude larger supersample.
  arXiv  Detail & Related papers  (2021-05-03T20:13:34Z)
• Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score [59.75318183140857]
  Estimating causal effects from randomized experiments is central to clinical research. Most methods for historical borrowing achieve reductions in variance by sacrificing strict type-I error rate control.
  arXiv  Detail & Related papers  (2020-12-17T21:10:10Z)
• Improving Maximum Likelihood Training for Text Generation with Density Ratio Estimation [51.091890311312085]
  We propose a new training scheme for auto-regressive sequence generative models, which is effective and stable when operating at large sample space encountered in text generation. Our method stably outperforms Maximum Likelihood Estimation and other state-of-the-art sequence generative models in terms of both quality and diversity.
  arXiv  Detail & Related papers  (2020-07-12T15:31:24Z)
• Balance-Subsampled Stable Prediction [55.13512328954456]
  We propose a novel balance-subsampled stable prediction (BSSP) algorithm based on the theory of fractional factorial design. A design-theoretic analysis shows that the proposed method can reduce the confounding effects among predictors induced by the distribution shift. Numerical experiments on both synthetic and real-world data sets demonstrate that our BSSP algorithm significantly outperforms the baseline methods for stable prediction across unknown test data.
  arXiv  Detail & Related papers  (2020-06-08T07:01:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.