Efficient Calibration for Decision Making

• URL: http://arxiv.org/abs/2511.13699v1
• Date: Mon, 17 Nov 2025 18:52:00 GMT
• Title: Efficient Calibration for Decision Making
• Authors: Parikshit Gopalan, Konstantinos Stavropoulos, Kunal Talwar, Pranay Tankala,
• Abstract summary: Hu and Wu (FOCS'24) use this to define an approximate calibration measure called calibration decision loss ($mathsfCDL$)<n>We develop a comprehensive theory of when $mathsfCDL_K$ is information-theoretically and computationally tractable.
• Score: 26.81026842833163
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A decision-theoretic characterization of perfect calibration is that an agent seeking to minimize a proper loss in expectation cannot improve their outcome by post-processing a perfectly calibrated predictor. Hu and Wu (FOCS'24) use this to define an approximate calibration measure called calibration decision loss ($\mathsf{CDL}$), which measures the maximal improvement achievable by any post-processing over any proper loss. Unfortunately, $\mathsf{CDL}$ turns out to be intractable to even weakly approximate in the offline setting, given black-box access to the predictions and labels. We suggest circumventing this by restricting attention to structured families of post-processing functions $K$. We define the calibration decision loss relative to $K$, denoted $\mathsf{CDL}_K$ where we consider all proper losses but restrict post-processings to a structured family $K$. We develop a comprehensive theory of when $\mathsf{CDL}_K$ is information-theoretically and computationally tractable, and use it to prove both upper and lower bounds for natural classes $K$. In addition to introducing new definitions and algorithmic techniques to the theory of calibration for decision making, our results give rigorous guarantees for some widely used recalibration procedures in machine learning.

Related papers

• Dimension-Free Decision Calibration for Nonlinear Loss Functions [27.879384242436835]
  calibration for high-dimensional prediction outcome spaces requires exponential computational and statistical complexity.<n>We introduce algorithms that, given $mathrmpoly(|A|,1/epsilon)$ samples, can efficiently post-process it to satisfy decision calibration without worsening accuracy.
  arXiv  Detail & Related papers  (2025-04-22T06:14:23Z)
• Smooth Calibration and Decision Making [11.51844809748468]
  We show that post-processing an online predictor with $eps$ to calibration achieves $O(sqrtepsilon)$ ECE and CDL.<n>The optimal bound is non-optimal compared with existing online calibration algorithms.
  arXiv  Detail & Related papers  (2025-04-22T04:55:41Z)
• Decision from Suboptimal Classifiers: Excess Risk Pre- and Post-Calibration [52.70324949884702]
  We quantify the excess risk incurred using approximate posterior probabilities in batch binary decision-making.<n>We identify regimes where recalibration alone addresses most of the regret, and regimes where the regret is dominated by the grouping loss.<n>On NLP experiments, we show that these quantities identify when the expected gain of more advanced post-training is worth the operational cost.
  arXiv  Detail & Related papers  (2025-03-23T10:52:36Z)
• Rethinking Early Stopping: Refine, Then Calibrate [49.966899634962374]
  We present a novel variational formulation of the calibration-refinement decomposition.<n>We provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training.
  arXiv  Detail & Related papers  (2025-01-31T15:03:54Z)
• Orthogonal Causal Calibration [55.28164682911196]
  We develop general algorithms for reducing the task of causal calibration to that of calibrating a standard (non-causal) predictive model.<n>Our results are exceedingly general, showing that essentially any existing calibration algorithm can be used in causal settings.
  arXiv  Detail & Related papers  (2024-06-04T03:35:25Z)
• Calibration Error for Decision Making [15.793486463552144]
  We propose a decision-theoretic calibration error, the Decision Loss (CDL), defined as the maximum improvement in decision payoff obtained by the predictions. We show separations between CDL and existing calibration error metrics, including the most well-studied metric Expected Error (ECE) Our main technical contribution is a new efficient algorithm for online calibration that achieves near-optimal $O(fraclog TsqrtT)$ expected CDL.
  arXiv  Detail & Related papers  (2024-04-21T01:53:20Z)
• Decision-Making under Miscalibration [14.762226638396209]
  ML-based predictions are used to inform consequential decisions about individuals. We formalize a natural (distribution-free) solution concept: given anticipated miscalibration of $alpha$, we propose using the threshold $j$ that minimizes the worst-case regret. We provide closed form expressions for $j$ when miscalibration is measured using both expected and maximum calibration error. We validate our theoretical findings on real data, demonstrating that there are natural cases in which making decisions using $j$ improves the clinical utility.
  arXiv  Detail & Related papers  (2022-03-18T10:44:11Z)
• Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration [118.26862029820447]
  We introduce a new notion -- emphdecision calibration -- that requires the predicted distribution and true distribution to be indistinguishable'' to a set of downstream decision-makers. Decision calibration improves decision-making on skin lesions and ImageNet classification with modern neural network.
  arXiv  Detail & Related papers  (2021-07-12T20:17:28Z)
• Localized Calibration: Metrics and Recalibration [133.07044916594361]
  We propose a fine-grained calibration metric that spans the gap between fully global and fully individualized calibration. We then introduce a localized recalibration method, LoRe, that improves the LCE better than existing recalibration methods.
  arXiv  Detail & Related papers  (2021-02-22T07:22:12Z)
• Post-hoc Calibration of Neural Networks by g-Layers [51.42640515410253]
  In recent years, there is a surge of research on neural network calibration. It is known that minimizing Negative Log-Likelihood (NLL) will lead to a calibrated network on the training set if the global optimum is attained. We prove that even though the base network ($f$) does not lead to the global optimum of NLL, by adding additional layers ($g$) and minimizing NLL by optimizing the parameters of $g$ one can obtain a calibrated network.
  arXiv  Detail & Related papers  (2020-06-23T07:55:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.