Condition-Stratified Robustness Analysis of Post-Hoc Calibration Methods for Probabilistic Classifiers

• URL: http://arxiv.org/abs/2607.11542v1
• Date: Mon, 13 Jul 2026 13:28:31 GMT
• Title: Condition-Stratified Robustness Analysis of Post-Hoc Calibration Methods for Probabilistic Classifiers
• Abstract summary: We compare temperature scaling (TEMP) and isotonic regression (ISO) across four controlled conditions.<n>Four hypothesis groups are evaluated: discrimination deltas with Holm-corrected control (H1), Brier score differences (H2), calibration slope outcomes (H3), and AUROC differences under best-condition setups (H4)
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Post-hoc calibration is widely adopted to correct probability estimates from trained classifiers, yet most evaluations report aggregate performance without testing whether that performance holds across distinct operating conditions within a single dataset. We present a pre-registered, condition-stratified robustness analysis comparing temperature scaling (TEMP) and isotonic regression (ISO) across four controlled conditions (C1--C4). Four hypothesis groups are evaluated: discrimination deltas with Holm-corrected multiplicity control (H1), Brier score differences (H2), calibration slope outcomes (H3), and AUROC differences under best-condition setups (H4). TEMP-minus-ISO discrimination deltas remain small across all conditions (-0.0155 to 0.0139), with Holm-adjusted p-values of 0.9895 everywhere. TEMP Brier differences are consistently negative (C1: -0.0002 through C4: -0.0074), while ISO shows sign reversals. TEMP calibration slopes stay closer to unity in every condition (range 0.7597--0.9493) than ISO slopes (0.1364--0.2726). AUROC differences shift from near zero in C1 (-0.0004) to positive in C4 (0.0264). These results establish that in-dataset robustness is condition-dependent and metric-specific. No claim of external transportability is made.

Related papers

• Semiparametric Inference for Conditional Shapley Feature Importance [0.0]
  This paper studies the conditional formulation, in which out-of-coalition features are integrated out under their true conditional distribution.<n>We propose a one-step estimator with K-fold cross-fitting and a U-statistic correction of the squared loss interval.<n>A Pinsker-type bound quantifies the bias from misspecifying the working copula class, while vine copulas keep conditional sampling tractable.
  arXiv  Detail & Related papers  (2026-09-09T15:19:24Z)
• Reliability-Aware Hybrid-K Ensemble Selection for Cervical Cytology Classification: Integrating Discrimination, Calibration, and Selective Prediction [3.093890460224435]
  This study proposes a reliability-aware Hybrid-K ensemble selection framework for multiclass cervical classification.<n>Nine deep learning architectures were evaluated using a fixed stratified five-fold partition and three training seeds.<n>The final Hybrid-2 ensemble, comprising Swin-Tiny and TinyViT-5M, reduced AURC by 43%, NLL by 17%, and WC-ECE by 36% relative to the best individual model.
  arXiv  Detail & Related papers  (2026-09-01T02:30:41Z)
• KAISEN: Reproducible Subgroup Fairness Auditing for Clinical Risk Models [0.0]
  KAISEN is a five-phase audit pipeline covering subgroup stratification, disparity measurement, mechanism diagnostics, post-hoc mitigation, and drift monitoring.<n>All results are synthetic with known ground truth and do not establish clinical validity.
  arXiv  Detail & Related papers  (2026-07-30T17:57:18Z)
• Mapping the Evaluation Frontier: An Empirical Survey of the Bias-Reliability Tradeoff Across Eleven Evaluator-Agent Conditions [1.0152838128195467]
  We extend the empirical base to 11 conditions, measuring gamma and H for all 11 conditions.<n>Conditions with low evaluator coupling (gamma 0.2) exhibit high measurement noise.<n>Conditions with strong coupling (gamma > 0.9) achieve low noise.<n>Four GPT-4o conditions show gamma=0.000 and H=1.000 across all seeds.
  arXiv  Detail & Related papers  (2026-07-01T01:04:33Z)
• Calibration, Uncertainty Communication, and Deployment Readiness in CKD Risk Prediction: A Framework Evaluation Study [0.0]
  Machine learning models for chronic kidney disease often post strong discrimination scores on internal test sets.<n>We trained five classifiers on the UCI CKD dataset (400 patients, 62.5% CKD prevalence)<n>We evaluated each across calibration quality, conformal prediction coverage, and an eight-criterion deployment readiness framework.
  arXiv  Detail & Related papers  (2026-05-20T17:12:48Z)
• PAIR-CI: Calibrated Conditional Independence Testing for Causal Discovery with Incomplete Data [0.0]
  We introduce PAIR-CI, a nonparametric conditional independence (CI) test that restores calibration by integrating multiple imputation directly into the inferential procedure.<n>A provably consistent variance estimator jointly accounts for uncertainty arising from cross-validation and multiple imputation.
  arXiv  Detail & Related papers  (2026-05-06T12:34:37Z)
• The Verification Tax: Fundamental Limits of AI Auditing in the Rare-Error Regime [0.0]
  Most cited calibration result in deep learning -- post-temperature-scaling ECE of 0.012 on CIFAR-100 -- is below the statistical noise floor.<n>We prove this is not a failure of the experiment but a law: the minimax rate for estimating calibration error with model error rate epsilon is Theta((Lepsilon/m)2/3), and no estimator can beat it.
  arXiv  Detail & Related papers  (2026-04-14T16:48:24Z)
• Robust Regression with Adaptive Contamination in Response: Optimal Rates and Computational Barriers [65.58071581164043]
  We study robust regression under a contamination model in which covariates are clean while the responses may be corrupted in an adaptive manner.<n>We show that the additional information can be carefully exploited to construct an estimator that a better estimation rate than achieves that attainable under Huber contamination.<n>Despite the improvement over the Huber model from an information-theoretic perspective, we provide formal evidence that the problem exhibits strong information-computation gaps.
  arXiv  Detail & Related papers  (2026-04-05T19:07:24Z)
• All Models Are Miscalibrated, But Some Less So: Comparing Calibration with Conditional Mean Operators [12.103487148356747]
  We propose a kernel calibration error based on the Hilbert-Schmidt norm of the difference between conditional mean operators.<n>Our experiments show that CKCE provides a more consistent ranking of models by their calibration error and is more robust against distribution shift.
  arXiv  Detail & Related papers  (2025-02-17T05:52:09Z)
• Robust Fine-tuning of Zero-shot Models via Variance Reduction [56.360865951192324]
  When fine-tuning zero-shot models, our desideratum is for the fine-tuned model to excel in both in-distribution (ID) and out-of-distribution (OOD) We propose a sample-wise ensembling technique that can simultaneously attain the best ID and OOD accuracy without the trade-offs.
  arXiv  Detail & Related papers  (2024-11-11T13:13:39Z)
• Orthogonal Causal Calibration [55.28164682911196]
  We develop general algorithms for reducing the task of causal calibration to that of calibrating a standard (non-causal) predictive model.<n>Our results are exceedingly general, showing that essentially any existing calibration algorithm can be used in causal settings.
  arXiv  Detail & Related papers  (2024-06-04T03:35:25Z)
• Controlling Moments with Kernel Stein Discrepancies [74.82363458321939]
  Kernel Stein discrepancies (KSDs) measure the quality of a distributional approximation.<n>We first show that standard KSDs used for weak convergence control fail to control moment convergence.<n>We then provide sufficient conditions under which alternative diffusion KSDs control both moment and weak convergence.
  arXiv  Detail & Related papers  (2022-11-10T08:24:52Z)
• Targeted Separation and Convergence with Kernel Discrepancies [61.973643031360254]
  kernel-based discrepancy measures are required to (i) separate a target P from other probability measures or (ii) control weak convergence to P.<n>In this article we derive new sufficient and necessary conditions to ensure (i) and (ii)<n>For MMDs on separable metric spaces, we characterize those kernels that separate Bochner embeddable measures and introduce simple conditions for separating all measures with unbounded kernels.
  arXiv  Detail & Related papers  (2022-09-26T16:41:16Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.