Another look at inference after prediction

• URL: http://arxiv.org/abs/2411.19908v2
• Date: Fri, 06 Dec 2024 15:34:08 GMT
• Title: Another look at inference after prediction
• Authors: Jessica Gronsbell, Jianhui Gao, Yaqi Shi, Zachary R. McCaw, David Cheng,
• Abstract summary: Prediction-based (PB) inference is increasingly used in applications where the outcome of interest is difficult to obtain.<n>We study the statistical efficiency of the prediction-powered inference (PPI) estimator.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Prediction-based (PB) inference is increasingly used in applications where the outcome of interest is difficult to obtain, but its predictors are readily available. Unlike traditional inference, PB inference performs statistical inference using a partially observed outcome and a set of covariates by leveraging a prediction of the outcome generated from a machine learning (ML) model. Motwani and Witten (2023) recently revisited two innovative PB inference approaches for ordinary least squares. They found that the method proposed by Wang et al. (2020) yields a consistent estimator for the association of interest when the ML model perfectly captures the underlying regression function. Conversely, the prediction-powered inference (PPI) method proposed by Angelopoulos et al. (2023) yields valid inference regardless of the model's accuracy. In this paper, we study the statistical efficiency of the PPI estimator. Our analysis reveals that a more efficient estimator, proposed 25 years ago by Chen and Chen (2000), can be obtained by simply adding a weight to the PPI estimator. We also contextualize PB inference with methods from the economics and statistics literature dating back to the 1960s. Our extensive theoretical and numerical analyses indicate that the Chen and Chen (CC) estimator offers a balance between robustness to ML model specification and statistical efficiency, making it the preferred choice for use in practice.

Related papers

• Prediction-Powered Adaptive Shrinkage Estimation [0.9208007322096532]
  Prediction-Powered Adaptive Shrinkage (PAS) is a method that bridges PPI with empirical Bayes shrinkage to improve the estimation of multiple means. PAS adapts to the reliability of the ML predictions and outperforms traditional and modern baselines in large-scale applications.
  arXiv  Detail & Related papers  (2025-02-20T00:24:05Z)
• FAB-PPI: Frequentist, Assisted by Bayes, Prediction-Powered Inference [0.0]
  Prediction-powered inference (PPI) enables valid statistical inference by combining experimental data with machine learning predictions. We propose to inform the PPI framework with prior knowledge on the quality of the predictions. The resulting method, which we call frequentist, assisted by Bayes, PPI (FAB-PPI), improves over PPI when the observed prediction quality is likely under the prior.
  arXiv  Detail & Related papers  (2025-02-04T14:46:08Z)
• Predictions as Surrogates: Revisiting Surrogate Outcomes in the Age of AI [12.569286058146343]
  We establish a formal connection between the decades-old surrogate outcome model in biostatistics and the emerging field of prediction-powered inference (PPI) We develop recalibrated prediction-powered inference, a more efficient approach to statistical inference than existing PPI proposals. We demonstrate significant gains in effective sample size over existing PPI proposals via three applications leveraging state-of-the-art machine learning/AI models.
  arXiv  Detail & Related papers  (2025-01-16T18:30:33Z)
• Preference-Based Multi-Agent Reinforcement Learning: Data Coverage and Algorithmic Techniques [65.55451717632317]
  We study Preference-Based Multi-Agent Reinforcement Learning (PbMARL) We identify the Nash equilibrium from a preference-only offline dataset in general-sum games. Our findings underscore the multifaceted approach required for PbMARL.
  arXiv  Detail & Related papers  (2024-09-01T13:14:41Z)
• Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation [62.2436697657307]
  Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data. We propose a method called Stratified Prediction-Powered Inference (StratPPI) We show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies.
  arXiv  Detail & Related papers  (2024-06-06T17:37:39Z)
• PPI++: Efficient Prediction-Powered Inference [31.403415618169433]
  We present PPI++: a methodology for estimation and inference based on a small labeled dataset and a typically much larger dataset of machine-learning predictions. The methods automatically adapt to the quality of available predictions, yielding easy-to-compute confidence sets. PPI++ builds on prediction-powered inference (PPI), which targets the same problem setting, improving its computational and statistical efficiency.
  arXiv  Detail & Related papers  (2023-11-02T17:59:04Z)
• Revisiting inference after prediction [0.0]
  We show that a method proposed by Angelopoulos et al. (2023) successfully controls the type 1 error rate and provides confidence intervals with correct nominal coverage. However, the method proposed by Wang et al. ( 2020) provides valid inference only under very strong conditions that rarely hold in practice.
  arXiv  Detail & Related papers  (2023-06-23T19:06:50Z)
• Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
  We propose a framework for efficient and effective counterfactual inference implemented with neural networks. The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data. Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
  arXiv  Detail & Related papers  (2023-06-09T08:30:51Z)
• Prediction-Oriented Bayesian Active Learning [51.426960808684655]
  Expected predictive information gain (EPIG) is an acquisition function that measures information gain in the space of predictions rather than parameters. EPIG leads to stronger predictive performance compared with BALD across a range of datasets and models.
  arXiv  Detail & Related papers  (2023-04-17T10:59:57Z)
• Prediction-Powered Inference [68.97619568620709]
  Prediction-powered inference is a framework for performing valid statistical inference when an experimental dataset is supplemented with predictions from a machine-learning system. The framework yields simple algorithms for computing provably valid confidence intervals for quantities such as means, quantiles, and linear and logistic regression coefficients. Prediction-powered inference could enable researchers to draw valid and more data-efficient conclusions using machine learning.
  arXiv  Detail & Related papers  (2023-01-23T18:59:28Z)
• Robust Output Analysis with Monte-Carlo Methodology [0.0]
  In predictive modeling with simulation or machine learning, it is critical to accurately assess the quality of estimated values. We propose a unified output analysis framework for simulation and machine learning outputs through the lens of Monte Carlo sampling.
  arXiv  Detail & Related papers  (2022-07-27T16:21:59Z)
• Excess risk analysis for epistemic uncertainty with application to variational inference [110.4676591819618]
  We present a novel EU analysis in the frequentist setting, where data is generated from an unknown distribution. We show a relation between the generalization ability and the widely used EU measurements, such as the variance and entropy of the predictive distribution. We propose new variational inference that directly controls the prediction and EU evaluation performances based on the PAC-Bayesian theory.
  arXiv  Detail & Related papers  (2022-06-02T12:12:24Z)
• When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting [70.54920804222031]
  Most existing forecasting models disregard uncertainty quantification, resulting in mis-calibrated predictions. Recent works in deep neural models for uncertainty-aware time-series forecasting also have several limitations. We model the forecasting task as a probabilistic generative process and propose a functional neural process model called EPIFNP.
  arXiv  Detail & Related papers  (2021-06-07T18:31:47Z)
• Double Robust Representation Learning for Counterfactual Prediction [68.78210173955001]
  We propose a novel scalable method to learn double-robust representations for counterfactual predictions. We make robust and efficient counterfactual predictions for both individual and average treatment effects. The algorithm shows competitive performance with the state-of-the-art on real world and synthetic data.
  arXiv  Detail & Related papers  (2020-10-15T16:39:26Z)
• Counterfactual Predictions under Runtime Confounding [74.90756694584839]
  We study the counterfactual prediction task in the setting where all relevant factors are captured in the historical data. We propose a doubly-robust procedure for learning counterfactual prediction models in this setting.
  arXiv  Detail & Related papers  (2020-06-30T15:49:05Z)
• Nonparametric inverse probability weighted estimators based on the highly adaptive lasso [0.966840768820136]
  Inparametric inverse probability weighted estimators are known to be inefficient and suffer from the curse of dimensionality. We propose a class of nonparametric inverse probability weighted estimators in which the weighting mechanism is estimated via undersmoothing of the highly adaptive lasso. Our developments have broad implications for the construction of efficient inverse probability weighted estimators in large statistical models and a variety of problem settings.
  arXiv  Detail & Related papers  (2020-05-22T17:49:46Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.