Do We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Data

• URL: http://arxiv.org/abs/2512.05456v1
• Date: Fri, 05 Dec 2025 06:24:23 GMT
• Title: Do We Really Even Need Data? A Modern Look at Drawing Inference with Predicted Data
• Authors: Stephen Salerno, Kentaro Hoffman, Awan Afiaz, Anna Neufeld, Tyler H. McCormick, Jeffrey T. Leek,
• Abstract summary: We show that high predictive accuracy does not guarantee valid downstream inference.<n>We show that all such failures reduce to statistical notions of (i) bias, when predictions systematically shift the estimand or distort relationships among variables, and (ii) variance, when uncertainty from the prediction model and the intrinsic variability of the true data are ignored.
• Score: 0.8415089854734883
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As artificial intelligence and machine learning tools become more accessible, and scientists face new obstacles to data collection (e.g., rising costs, declining survey response rates), researchers increasingly use predictions from pre-trained algorithms as substitutes for missing or unobserved data. Though appealing for financial and logistical reasons, using standard tools for inference can misrepresent the association between independent variables and the outcome of interest when the true, unobserved outcome is replaced by a predicted value. In this paper, we characterize the statistical challenges inherent to drawing inference with predicted data (IPD) and show that high predictive accuracy does not guarantee valid downstream inference. We show that all such failures reduce to statistical notions of (i) bias, when predictions systematically shift the estimand or distort relationships among variables, and (ii) variance, when uncertainty from the prediction model and the intrinsic variability of the true data are ignored. We then review recent methods for conducting IPD and discuss how this framework is deeply rooted in classical statistical theory. We then comment on some open questions and interesting avenues for future work in this area, and end with some comments on how to use predicted data in scientific studies that is both transparent and statistically principled.

Related papers

• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Prediction-Powered Inference with Imputed Covariates and Nonuniform Sampling [20.078602767179355]
  Failure to properly account for errors in machine learning predictions renders standard statistical procedures invalid.<n>We introduce bootstrap confidence intervals that apply when the complete data is a nonuniform (i.e., weighted, stratified, or clustered) sample and to settings where an arbitrary subset of features is imputed.<n>We prove that these confidence intervals are valid under no assumptions on the quality of the machine learning model and are no wider than the intervals obtained by methods that do not use machine learning predictions.
  arXiv  Detail & Related papers  (2025-01-30T18:46:43Z)
• Do We Really Even Need Data? [2.3749120526936465]
  Researchers increasingly use predictions from pre-trained algorithms as outcome variables. Standard tools for inference can misrepresent the association between independent variables and the outcome of interest when the true, unobserved outcome is replaced by a predicted value.
  arXiv  Detail & Related papers  (2024-01-14T23:19:21Z)
• Cross-Prediction-Powered Inference [15.745692520785074]
  Cross-prediction is a method for valid inference powered by machine learning. We show that cross-prediction is consistently more powerful than an adaptation of prediction-powered inference.
  arXiv  Detail & Related papers  (2023-09-28T17:01:58Z)
• Prediction-Powered Inference [68.97619568620709]
  Prediction-powered inference is a framework for performing valid statistical inference when an experimental dataset is supplemented with predictions from a machine-learning system. The framework yields simple algorithms for computing provably valid confidence intervals for quantities such as means, quantiles, and linear and logistic regression coefficients. Prediction-powered inference could enable researchers to draw valid and more data-efficient conclusions using machine learning.
  arXiv  Detail & Related papers  (2023-01-23T18:59:28Z)
• D-BIAS: A Causality-Based Human-in-the-Loop System for Tackling Algorithmic Bias [57.87117733071416]
  We propose D-BIAS, a visual interactive tool that embodies human-in-the-loop AI approach for auditing and mitigating social biases. A user can detect the presence of bias against a group by identifying unfair causal relationships in the causal network. For each interaction, say weakening/deleting a biased causal edge, the system uses a novel method to simulate a new (debiased) dataset.
  arXiv  Detail & Related papers  (2022-08-10T03:41:48Z)
• Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictions [59.284907093349425]
  Large amounts of training data are one of the major reasons for the high performance of state-of-the-art NLP models. We provide a language for describing how training data influences predictions, through a causal framework. Our framework bypasses the need to retrain expensive models and allows us to estimate causal effects based on observational data alone.
  arXiv  Detail & Related papers  (2022-07-28T17:36:24Z)
• Conformal Prediction Under Feedback Covariate Shift for Biomolecular Design [56.86533144730384]
  We introduce a method to quantify predictive uncertainty in settings where the training and test data are statistically dependent.<n>As a motivating use case, we demonstrate with several real data sets how our method quantifies uncertainty for the predicted fitness of designed proteins.
  arXiv  Detail & Related papers  (2022-02-08T02:59:12Z)
• Learning to Predict Trustworthiness with Steep Slope Loss [69.40817968905495]
  We study the problem of predicting trustworthiness on real-world large-scale datasets. We observe that the trustworthiness predictors trained with prior-art loss functions are prone to view both correct predictions and incorrect predictions to be trustworthy. We propose a novel steep slope loss to separate the features w.r.t. correct predictions from the ones w.r.t. incorrect predictions by two slide-like curves that oppose each other.
  arXiv  Detail & Related papers  (2021-09-30T19:19:09Z)
• Double Robust Representation Learning for Counterfactual Prediction [68.78210173955001]
  We propose a novel scalable method to learn double-robust representations for counterfactual predictions. We make robust and efficient counterfactual predictions for both individual and average treatment effects. The algorithm shows competitive performance with the state-of-the-art on real world and synthetic data.
  arXiv  Detail & Related papers  (2020-10-15T16:39:26Z)
• Robust Validation: Confident Predictions Even When Distributions Shift [19.327409270934474]
  We describe procedures for robust predictive inference, where a model provides uncertainty estimates on its predictions rather than point predictions. We present a method that produces prediction sets (almost exactly) giving the right coverage level for any test distribution in an $f$-divergence ball around the training population. An essential component of our methodology is to estimate the amount of expected future data shift and build robustness to it.
  arXiv  Detail & Related papers  (2020-08-10T17:09:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.