Testable and Actionable Calibration for Full Swap Regret

• URL: http://arxiv.org/abs/2605.17749v1
• Date: Mon, 18 May 2026 02:04:08 GMT
• Title: Testable and Actionable Calibration for Full Swap Regret
• Abstract summary: We introduce a new calibration measure, Soft-Binned Decision Loss (SCDL), which we prove is fully actionable without weakening either requirement.<n>SCDL satisfies other desired properties such as continuity and consistency.
• Score: 14.699455652461724
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: AI generated predictions increasingly inform decision making in critical tasks, and therefore must be trustworthy. One widely used measure of trustworthiness is calibration, which requires that the predictions match the true frequencies and can be treated like real probabilities of a given outcome. However, defining calibration is subtle, and designing good measures of calibration error has been an active topic of recent research. The first goal is to find calibration measures that are actionable, meaning they can inform decision makers about their utility loss when predictions are treated as true probabilities, which is known as swap regret. The second goal is to find calibration measures that are testable, meaning that calibration error can be measured from a small sample of predictions and outcomes. Although these are very basic requirements, there is no existing calibration measure that fully satisfies both properties, and all existing measures relax actionability by bounding a weaker notion of swap regret, or relax testability by having suboptimal estimation error. We introduce a new calibration measure, Soft-Binned Calibration Decision Loss (SCDL), which we prove is fully actionable without weakening either requirement, and testable with nearly optimal error rate. In addition, SCDL satisfies other desired properties such as continuity and consistency. We also provide a set of experiments confirming that the theoretical advantages of SCDL compared to other measures lead to better performance in practice.

Related papers

• Robust Decision Making with Partially Calibrated Forecasts [32.96842403002962]
  We study how a conservative decision maker should map predictions endowed with weaker (partial'') calibration guarantees to actions.<n>We characterize their minimax optimal decision rule via a duality argument, and show that surprisingly, trusting the predictions and acting accordingly'' is recovered in this minimax sense.
  arXiv  Detail & Related papers  (2025-10-27T16:09:07Z)
• Making and Evaluating Calibrated Forecasts [10.153382419318023]
  We introduce a perfectly truthful calibration measure for multi-class prediction tasks.<n>We mathematically prove and empirically verify that our calibration measure exhibits superior robustness.<n>This result addresses the non-robustness issue of binned ECE.
  arXiv  Detail & Related papers  (2025-10-07T19:11:03Z)
• A Perfectly Truthful Calibration Measure [9.916245963243542]
  A calibration measure quantifies how far a predictor is from perfect calibration.<n>We design a simple, perfectly and strictly truthful, sound and complete calibration measure in the batch setting.
  arXiv  Detail & Related papers  (2025-08-18T17:09:34Z)
• A Confidence Interval for the $\ell_2$ Expected Calibration Error [35.88784957918326]
  We develop confidence intervals $ell$ Expected the Error (ECE)<n>We consider top-1-to-$k$ calibration, which includes both the popular notion of confidence calibration as well as calibration.<n>For a debiased estimator of the ECE, we show normality, but with different convergence rates and variances for calibrated and misd models.
  arXiv  Detail & Related papers  (2024-08-16T20:00:08Z)
• Truthfulness of Calibration Measures [18.21682539787221]
  A calibration measure is said to be truthful if the forecaster minimizes expected penalty by predicting the conditional expectation of the next outcome. This makes it an essential desideratum for calibration measures, alongside typical requirements, such as soundness and completeness. We introduce a new calibration measure termed the Subsampled Smooth Error (SSCE) under which truthful prediction is optimal up to a constant multiplicative factor.
  arXiv  Detail & Related papers  (2024-07-19T02:07:55Z)
• Orthogonal Causal Calibration [55.28164682911196]
  We develop general algorithms for reducing the task of causal calibration to that of calibrating a standard (non-causal) predictive model.<n>Our results are exceedingly general, showing that essentially any existing calibration algorithm can be used in causal settings.
  arXiv  Detail & Related papers  (2024-06-04T03:35:25Z)
• Towards Certification of Uncertainty Calibration under Adversarial Attacks [96.48317453951418]
  We show that attacks can significantly harm calibration, and thus propose certified calibration as worst-case bounds on calibration under adversarial perturbations.<n>We propose novel calibration attacks and demonstrate how they can improve model calibration through textitadversarial calibration training
  arXiv  Detail & Related papers  (2024-05-22T18:52:09Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• T-Cal: An optimal test for the calibration of predictive models [49.11538724574202]
  We consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. We propose T-Cal, a minimax test for calibration based on a de-biased plug-in estimator of the $ell$-Expected Error (ECE)
  arXiv  Detail & Related papers  (2022-03-03T16:58:54Z)
• Localized Calibration: Metrics and Recalibration [133.07044916594361]
  We propose a fine-grained calibration metric that spans the gap between fully global and fully individualized calibration. We then introduce a localized recalibration method, LoRe, that improves the LCE better than existing recalibration methods.
  arXiv  Detail & Related papers  (2021-02-22T07:22:12Z)
• Unsupervised Calibration under Covariate Shift [92.02278658443166]
  We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
  arXiv  Detail & Related papers  (2020-06-29T21:50:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.