Fast and Scalable Score-Based Kernel Calibration Tests

• URL: http://arxiv.org/abs/2510.14711v1
• Date: Thu, 16 Oct 2025 14:11:14 GMT
• Title: Fast and Scalable Score-Based Kernel Calibration Tests
• Abstract summary: We introduce the Kernel Conditional Stein Discrepancy test (KCCSD test), a non-parametric, kernel-based test for assessing the calibration of probabilistic models with well-defined scores.<n>In contrast to previous methods, our test avoids the need for possibly expensive expectation approximations while providing control over its type-I error.
• Score: 28.41894759475933
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce the Kernel Calibration Conditional Stein Discrepancy test (KCCSD test), a non-parametric, kernel-based test for assessing the calibration of probabilistic models with well-defined scores. In contrast to previous methods, our test avoids the need for possibly expensive expectation approximations while providing control over its type-I error. We achieve these improvements by using a new family of kernels for score-based probabilities that can be estimated without probability density samples, and by using a conditional goodness-of-fit criterion for the KCCSD test's U-statistic. We demonstrate the properties of our test on various synthetic settings.

Related papers

• Empirically Calibrated Conditional Independence Tests [2.28126966226433]
  Conditional independence tests (CIT) are widely used for causal discovery and feature selection.<n>We propose Empirically Calibrated Conditional Independence Tests (ECCIT) to measure and correct for miscalibration.
  arXiv  Detail & Related papers  (2026-02-24T15:56:19Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Kernel conditional tests from learning-theoretic bounds [16.813275168865953]
  We propose a framework for hypothesis testing on conditional probability distributions.<n>We then use to construct statistical tests of functionals of conditional distributions.<n>Our results establish a comprehensive foundation for conditional testing on functionals.
  arXiv  Detail & Related papers  (2025-06-04T12:53:13Z)
• Robust Kernel Hypothesis Testing under Data Corruption [6.430258446597413]
  We propose a general method for constructing robust permutation tests under data corruption.<n>We prove their consistency in power under minimal conditions.<n>This contributes to the practical deployment of hypothesis tests for real-world applications with potential adversarial attacks.
  arXiv  Detail & Related papers  (2024-05-30T10:23:16Z)
• Minimax Optimal Goodness-of-Fit Testing with Kernel Stein Discrepancy [13.429541377715298]
  We explore the minimax optimality of goodness-of-fit tests on general domains using the kernelized Stein discrepancy (KSD)<n>KSD is a powerful framework for goodness-of-fit testing, but its statistical optimality remains largely unexplored.<n>We introduce an adaptive test capable of achieving minimax optimality up to a logarithmic factor by adapting to unknown parameters.
  arXiv  Detail & Related papers  (2024-04-12T07:06:12Z)
• Practical Kernel Tests of Conditional Independence [33.275712245547815]
  SplitKCI is an automated method for bias control for the Kernel-based Conditional Independence test based on data splitting.<n>We show that our approach significantly improves test level control for KCI without sacrificing test power.
  arXiv  Detail & Related papers  (2024-02-20T18:07:59Z)
• Precise Error Rates for Computationally Efficient Testing [67.30044609837749]
  We revisit the question of simple-versus-simple hypothesis testing with an eye towards computational complexity.<n>An existing test based on linear spectral statistics achieves the best possible tradeoff curve between type I and type II error rates.
  arXiv  Detail & Related papers  (2023-11-01T04:41:16Z)
• Sequential Kernelized Independence Testing [77.237958592189]
  We design sequential kernelized independence tests inspired by kernelized dependence measures.<n>We demonstrate the power of our approaches on both simulated and real data.
  arXiv  Detail & Related papers  (2022-12-14T18:08:42Z)
• Boost Test-Time Performance with Closed-Loop Inference [85.43516360332646]
  We propose to predict hard-classified test samples in a looped manner to boost the model performance. We first devise a filtering criterion to identify those hard-classified test samples that need additional inference loops. For each hard sample, we construct an additional auxiliary learning task based on its original top-$K$ predictions to calibrate the model.
  arXiv  Detail & Related papers  (2022-03-21T10:20:21Z)
• T-Cal: An optimal test for the calibration of predictive models [49.11538724574202]
  We consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. We propose T-Cal, a minimax test for calibration based on a de-biased plug-in estimator of the $ell$-Expected Error (ECE)
  arXiv  Detail & Related papers  (2022-03-03T16:58:54Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)
• Learning Kernel Tests Without Data Splitting [18.603394415852765]
  We propose an approach that enables learning the hyper parameters and testing on the full sample without data splitting. Our approach's test power is empirically larger than that of the data-splitting approach, regardless of its split proportion.
  arXiv  Detail & Related papers  (2020-06-03T14:07:39Z)
• A Kernel Stein Test for Comparing Latent Variable Models [48.32146056855925]
  We propose a kernel-based nonparametric test of relative goodness of fit, where the goal is to compare two models, both of which may have unobserved latent variables. We show that our test significantly outperforms the relative Maximum Mean Discrepancy test, which is based on samples from the models and does not exploit the latent structure.
  arXiv  Detail & Related papers  (2019-07-01T07:46:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.