Calibration without labels in multiple testing

• URL: http://arxiv.org/abs/2606.19737v1
• Date: Thu, 18 Jun 2026 03:06:11 GMT
• Title: Calibration without labels in multiple testing
• Abstract summary: Large-scale hypothesis testing supports probability claims about individual hypotheses, as in empirical Bayes methods for estimating local false discovery rates.<n>We study how such claims can be interpreted as approximately calibrated forecasts of the null hypothesis, yielding interpretable error probabilities even under model misspecification.<n>We address this challenge by constructing a set of pseudo-labels, derived from the spacings of ordered $p$-values, which have the local false discovery rate as their regression target.
• Score: 2.435006380732194
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large-scale hypothesis testing supports probability claims about individual hypotheses, as in empirical Bayes methods for estimating local false discovery rates. We study how such claims can be interpreted as approximately calibrated forecasts of the null hypothesis, yielding interpretable error probabilities even under model misspecification. Our approach draws conceptual inspiration from probabilistic forecasting but addresses a different challenge: unlike forecasting, where labels are eventually observed, in multiple testing the ground truth is never revealed, so calibration must be assessed stochastically and established indirectly. We address this challenge by constructing a set of pseudo-labels, derived from the spacings of ordered $p$-values, which have the local false discovery rate as their regression target. Our construction unlocks existing tools for assessing and performing post-hoc calibration in multiple testing. Notably, we find on a large-scale empirical survey of published psychology and neuroscience literature that the $q$-value, a popular error measure based on the false discovery rate, can be severely miscalibrated.

Related papers

• Spiking the training data to correct for test set contamination [28.940486760749025]
  We propose to spike the training data by intentionally contaminating some test examples at known rates.<n>The spiked examples can then be used to calibrate predictors of model memorization which enable principled statistical correction of inflated test scores.
  arXiv  Detail & Related papers  (2026-05-24T02:06:59Z)
• Hypothesis Testing over Observable Regimes in Singular Models [0.12183405753834557]
  We show that the fundamental obstruction to testing in singular statistical models is not singularity itself, but the formulation of hypotheses on non-identifiable parameter quantities.<n>We formalize this overlap obstruction and show that hypotheses depending on non-identifiable parameter functions necessarily fail in this sense.<n>In contrast, hypotheses formulated over identifiable observables-quantities that are determined by the induced distribution-reduce entirely to classical testing theory.
  arXiv  Detail & Related papers  (2026-02-27T16:44:29Z)
• Predicting fixed-sample test decisions enables anytime-valid inference [0.3222802562733787]
  We introduce a simple procedure that transforms any fixed-sample hypothesis test into an anytime-valid test.<n>We ensure Type-I error control and near-optimal power with substantial sample savings when the null hypothesis is false.<n>In areas such as clinical trials, stopping early and safely can ensure that subjects receive the best treatments and accelerate the development of effective therapies.
  arXiv  Detail & Related papers  (2026-02-14T20:17:51Z)
• Pre-validation Revisited [79.92204034170092]
  We show properties and benefits of pre-validation in prediction, inference and error estimation by simulations and applications.<n>We propose not only an analytical distribution of the test statistic for the pre-validated predictor under certain models, but also a generic bootstrap procedure to conduct inference.
  arXiv  Detail & Related papers  (2025-05-21T00:20:14Z)
• Revisiting Confidence Estimation: Towards Reliable Failure Prediction [53.79160907725975]
  We find a general, widely existing but actually-neglected phenomenon that most confidence estimation methods are harmful for detecting misclassification errors. We propose to enlarge the confidence gap by finding flat minima, which yields state-of-the-art failure prediction performance.
  arXiv  Detail & Related papers  (2024-03-05T11:44:14Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Calibration tests beyond classification [30.616624345970973]
  Most supervised machine learning tasks are subject to irreducible prediction errors. Probabilistic predictive models address this limitation by providing probability distributions that represent a belief over plausible targets. Calibrated models guarantee that the predictions are neither over- nor under-confident.
  arXiv  Detail & Related papers  (2022-10-21T09:49:57Z)
• T-Cal: An optimal test for the calibration of predictive models [49.11538724574202]
  We consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. We propose T-Cal, a minimax test for calibration based on a de-biased plug-in estimator of the $ell$-Expected Error (ECE)
  arXiv  Detail & Related papers  (2022-03-03T16:58:54Z)
• Tracking disease outbreaks from sparse data with Bayesian inference [55.82986443159948]
  The COVID-19 pandemic provides new motivation for estimating the empirical rate of transmission during an outbreak. Standard methods struggle to accommodate the partial observability and sparse data common at finer scales. We propose a Bayesian framework which accommodates partial observability in a principled manner.
  arXiv  Detail & Related papers  (2020-09-12T20:37:33Z)
• Individual Calibration with Randomized Forecasting [116.2086707626651]
  We show that calibration for individual samples is possible in the regression setup if the predictions are randomized. We design a training objective to enforce individual calibration and use it to train randomized regression functions.
  arXiv  Detail & Related papers  (2020-06-18T05:53:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.