Prediction-Powered Inference with Imputed Covariates and Nonuniform Sampling

• URL: http://arxiv.org/abs/2501.18577v1
• Date: Thu, 30 Jan 2025 18:46:43 GMT
• Title: Prediction-Powered Inference with Imputed Covariates and Nonuniform Sampling
• Authors: Dan M. Kluger, Kerri Lu, Tijana Zrnic, Sherrie Wang, Stephen Bates,
• Abstract summary: Failure to properly account for errors in machine learning predictions renders standard statistical procedures invalid. We introduce bootstrap confidence intervals that apply when the complete data is a nonuniform (i.e., weighted, stratified, or clustered) sample and to settings where an arbitrary subset of features is imputed. We prove that these confidence intervals are valid under no assumptions on the quality of the machine learning model and are no wider than the intervals obtained by methods that do not use machine learning predictions.
• Score: 20.078602767179355
• License:
• Abstract: Machine learning models are increasingly used to produce predictions that serve as input data in subsequent statistical analyses. For example, computer vision predictions of economic and environmental indicators based on satellite imagery are used in downstream regressions; similarly, language models are widely used to approximate human ratings and opinions in social science research. However, failure to properly account for errors in the machine learning predictions renders standard statistical procedures invalid. Prior work uses what we call the Predict-Then-Debias estimator to give valid confidence intervals when machine learning algorithms impute missing variables, assuming a small complete sample from the population of interest. We expand the scope by introducing bootstrap confidence intervals that apply when the complete data is a nonuniform (i.e., weighted, stratified, or clustered) sample and to settings where an arbitrary subset of features is imputed. Importantly, the method can be applied to many settings without requiring additional calculations. We prove that these confidence intervals are valid under no assumptions on the quality of the machine learning model and are no wider than the intervals obtained by methods that do not use machine learning predictions.

Related papers

• Adversarial Debiasing for Unbiased Parameter Recovery [0.8749675983608172]
  We show how prediction errors from machine learning models can lead to bias in the estimates of regression coefficients. We propose a test for detecting bias, and demonstrate the use of an adversarial machine learning algorithm in order to de-bias predictions. We conduct simulations and empirical exercises using ground truth and satellite data on forest cover in Africa.
  arXiv  Detail & Related papers  (2025-02-17T20:54:56Z)
• Fair Generalized Linear Mixed Models [0.0]
  Fairness in machine learning aims to ensure that biases in the data and model inaccuracies do not lead to discriminatory decisions. We present an algorithm that can handle both problems simultaneously.
  arXiv  Detail & Related papers  (2024-05-15T11:42:41Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Prediction-Powered Inference [68.97619568620709]
  Prediction-powered inference is a framework for performing valid statistical inference when an experimental dataset is supplemented with predictions from a machine-learning system. The framework yields simple algorithms for computing provably valid confidence intervals for quantities such as means, quantiles, and linear and logistic regression coefficients. Prediction-powered inference could enable researchers to draw valid and more data-efficient conclusions using machine learning.
  arXiv  Detail & Related papers  (2023-01-23T18:59:28Z)
• Conformal prediction for the design problem [72.14982816083297]
  In many real-world deployments of machine learning, we use a prediction algorithm to choose what data to test next. In such settings, there is a distinct type of distribution shift between the training and test data. We introduce a method to quantify predictive uncertainty in such settings.
  arXiv  Detail & Related papers  (2022-02-08T02:59:12Z)
• Uncertainty Prediction for Machine Learning Models of Material Properties [0.0]
  Uncertainty in AI-based predictions of material properties is of immense importance for the success and reliability of AI applications in material science. We compare 3 different approaches to obtain such individual uncertainty, testing them on 12 ML-physical properties.
  arXiv  Detail & Related papers  (2021-07-16T16:33:55Z)
• A Note on High-Probability versus In-Expectation Guarantees of Generalization Bounds in Machine Learning [95.48744259567837]
  Statistical machine learning theory often tries to give generalization guarantees of machine learning models. Statements made about the performance of machine learning models have to take the sampling process into account. We show how one may transform one statement to another.
  arXiv  Detail & Related papers  (2020-10-06T09:41:35Z)
• Individual Calibration with Randomized Forecasting [116.2086707626651]
  We show that calibration for individual samples is possible in the regression setup if the predictions are randomized. We design a training objective to enforce individual calibration and use it to train randomized regression functions.
  arXiv  Detail & Related papers  (2020-06-18T05:53:10Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.