Prediction-Powered Active Testing

• URL: http://arxiv.org/abs/2607.08347v1
• Date: Thu, 09 Jul 2026 10:52:24 GMT
• Title: Prediction-Powered Active Testing
• Abstract summary: Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled.<n>We propose bfbfPrediction--Powered Active Testing (PPAT), a novel label--efficient risk estimation framework.
• Score: 14.083979525233316
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Active testing provides a label--efficient approach to risk estimation by adaptively selecting which test points should be labelled. However, existing estimators fail to exploit the informative predictions of powerful black--box models, even though such predictions are increasingly available in settings where labels remain expensive. To address this, we propose \textbf{Prediction--Powered Active Testing (PPAT)}, a novel label--efficient risk estimation framework that combines the unbiased LURE estimator \citep{farquhar2021statistical} with a prediction--powered control variate. Rather than using proxy predictions as biased pseudo--labels, PPAT uses them to residualise the loss, preserving unbiasedness while reducing variance. Beyond the estimator itself, PPAT also changes which points should be acquired: we derive oracle and practical surrogate--based acquisition rules tailored to reducing the variance of our estimator. Moreover, we establish asymptotic normality for PPAT, yielding asymptotically valid confidence intervals and thus a principled estimate of the uncertainty around our estimates. Across tabular regression and image--classification tasks, PPAT outperforms existing methods in risk estimation, while its confidence intervals attain the target coverage with substantially fewer labels and smaller widths.

Related papers

• Heuristic Pathologies and Further Variance Reduction via Uncertainty Propagation in the AIVAT Family of Techniques [62.94836578759359]
  We show how the uncertainty can be propagated to quantify the uncertainty of AIVAT estimates.<n>In our experiments, we use a dataset of 10,000 poker hands to demonstrate our pathology and uncertainty results.
  arXiv  Detail & Related papers  (2026-05-14T02:04:26Z)
• Statistical Inference for Score Decompositions [0.0]
  We introduce methods for score decompositions, which partition scoring functions for predictive assessment into three interpretable components.<n>We find that for survey inflation forecasts, discrimination abilities can differ significantly even when overall predictive ability does not.<n>By disentangling forecast accuracy from backtest performance, the method exposes critical shortcomings in current banking regulation.
  arXiv  Detail & Related papers  (2026-03-04T16:57:08Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• TRUST: Test-time Resource Utilization for Superior Trustworthiness [15.031121920821109]
  We propose a novel test-time optimization method that accounts for the impact of such noise to produce more reliable confidence estimates.<n>This score defines a monotonic subset-selection function, where population accuracy consistently increases as samples with lower scores are removed.
  arXiv  Detail & Related papers  (2025-06-06T12:52:32Z)
• TULiP: Test-time Uncertainty Estimation via Linearization and Weight Perturbation [11.334867025651233]
  We propose TULiP, a theoretically-driven uncertainty estimator for OOD detection.<n>Our approach considers a hypothetical perturbation applied to the network before convergence.<n>Our method exhibits state-of-the-art performance, particularly for near-distribution samples.
  arXiv  Detail & Related papers  (2025-05-22T17:16:41Z)
• OT Score: An OT based Confidence Score for Source Free Unsupervised Domain Adaptation [2.6912673131004468]
  We introduce the Optimal Transport (OT) score, a confidence metric derived from a novel theoretical analysis.<n> OT score is intuitively interpretable and theoretically rigorous.<n>It provides principled uncertainty estimates for any given set of target pseudo-labels.<n>It improves SFUDA performance through training-time reweighting and provides a reliable, label-free proxy for model performance.
  arXiv  Detail & Related papers  (2025-05-16T20:09:05Z)
• Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [65.21599711087538]
  Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample.<n>Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications.<n>We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
  arXiv  Detail & Related papers  (2024-03-18T05:49:45Z)
• Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization [59.758009422067]
  We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. We propose a new uncertainty Bellman equation (UBE) whose solution converges to the true posterior variance over values. We introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC) that can be applied for either risk-seeking or risk-averse policy optimization.
  arXiv  Detail & Related papers  (2023-12-07T15:55:58Z)
• Distribution-free uncertainty quantification for classification under label shift [105.27463615756733]
  We focus on uncertainty quantification (UQ) for classification problems via two avenues. We first argue that label shift hurts UQ, by showing degradation in coverage and calibration. We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
  arXiv  Detail & Related papers  (2021-03-04T20:51:03Z)
• DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
  Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner. We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
  arXiv  Detail & Related papers  (2021-02-16T23:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.