Calibration through the Lens of Indistinguishability

• URL: http://arxiv.org/abs/2509.02279v1
• Date: Tue, 02 Sep 2025 13:01:43 GMT
• Title: Calibration through the Lens of Indistinguishability
• Abstract summary: The study of calibration has seen a surge of recent interest, given the ubiquity of probabilistic predictions in machine learning.<n>This survey describes recent work on the foundational questions of how to define and measure calibration error.<n>A unifying viewpoint that emerges is that of calibration as a form of indistinguishability, between the world hypothesized by the predictor and the real world.
• Score: 11.288403109735542
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Calibration is a classical notion from the forecasting literature which aims to address the question: how should predicted probabilities be interpreted? In a world where we only get to observe (discrete) outcomes, how should we evaluate a predictor that hypothesizes (continuous) probabilities over possible outcomes? The study of calibration has seen a surge of recent interest, given the ubiquity of probabilistic predictions in machine learning. This survey describes recent work on the foundational questions of how to define and measure calibration error, and what these measures mean for downstream decision makers who wish to use the predictions to make decisions. A unifying viewpoint that emerges is that of calibration as a form of indistinguishability, between the world hypothesized by the predictor and the real world (governed by nature or the Bayes optimal predictor). In this view, various calibration measures quantify the extent to which the two worlds can be told apart by certain classes of distinguishers or statistical measures.

Related papers

• Making and Evaluating Calibrated Forecasts [10.153382419318023]
  We introduce a perfectly truthful calibration measure for multi-class prediction tasks.<n>We mathematically prove and empirically verify that our calibration measure exhibits superior robustness.<n>This result addresses the non-robustness issue of binned ECE.
  arXiv  Detail & Related papers  (2025-10-07T19:11:03Z)
• Measuring Informativeness Gap of (Mis)Calibrated Predictors [15.651406777700517]
  In many applications, decision-makers must choose between multiple predictive models that may all be miscalibrated.<n>Our framework strictly generalizes U-Calibration [KLST-23] and Decision Loss [HW-24], which compare a miscalibrated predictor to its calibrated counterpart.<n>Our second contribution is a dual characterization of the informativeness gap, which gives rise to a natural informativeness measure.
  arXiv  Detail & Related papers  (2025-07-16T10:01:22Z)
• Performative Prediction on Games and Mechanism Design [69.7933059664256]
  We study a collective risk dilemma where agents decide whether to trust predictions based on past accuracy.<n>As predictions shape collective outcomes, social welfare arises naturally as a metric of concern.<n>We show how to achieve better trade-offs and use them for mechanism design.
  arXiv  Detail & Related papers  (2024-08-09T16:03:44Z)
• Truthfulness of Calibration Measures [18.21682539787221]
  A calibration measure is said to be truthful if the forecaster minimizes expected penalty by predicting the conditional expectation of the next outcome. This makes it an essential desideratum for calibration measures, alongside typical requirements, such as soundness and completeness. We introduce a new calibration measure termed the Subsampled Smooth Error (SSCE) under which truthful prediction is optimal up to a constant multiplicative factor.
  arXiv  Detail & Related papers  (2024-07-19T02:07:55Z)
• Boldness-Recalibration for Binary Event Predictions [0.0]
  Ideally, probability predictions are (i) well calibrated, (ii) accurate, and (iii) bold, i.e., spread out enough to be informative for decision making. There is a fundamental tension between calibration and boldness, since calibration metrics can be high when predictions are overly cautious, i.e., non-bold. The purpose of this work is to develop a Bayesian model selection-based approach to assess calibration, and a strategy for boldness-recalibration.
  arXiv  Detail & Related papers  (2023-05-05T18:14:47Z)
• Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation [51.809741427975105]
  This work aims to assess how well a model performs under distribution shifts without using labels. We use the nuclear norm that has been shown to be effective in characterizing both properties. We show that the nuclear norm is more accurate and robust in accuracy than existing methods.
  arXiv  Detail & Related papers  (2023-02-02T13:30:48Z)
• Calibration tests beyond classification [30.616624345970973]
  Most supervised machine learning tasks are subject to irreducible prediction errors. Probabilistic predictive models address this limitation by providing probability distributions that represent a belief over plausible targets. Calibrated models guarantee that the predictions are neither over- nor under-confident.
  arXiv  Detail & Related papers  (2022-10-21T09:49:57Z)
• Predictive Inference with Feature Conformal Prediction [80.77443423828315]
  We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces. From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions. Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
  arXiv  Detail & Related papers  (2022-10-01T02:57:37Z)
• On the Calibration of Probabilistic Classifier Sets [6.759124697337311]
  We extend the notion of calibration to evaluate the validity of an aleatoric uncertainty representation. We show that ensembles of deep neural networks are often not well calibrated.
  arXiv  Detail & Related papers  (2022-05-20T10:57:46Z)
• Dense Uncertainty Estimation [62.23555922631451]
  In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation. We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
  arXiv  Detail & Related papers  (2021-10-13T01:23:48Z)
• When Does Uncertainty Matter?: Understanding the Impact of Predictive Uncertainty in ML Assisted Decision Making [68.19284302320146]
  We carry out user studies to assess how people with differing levels of expertise respond to different types of predictive uncertainty. We found that showing posterior predictive distributions led to smaller disagreements with the ML model's predictions. This suggests that posterior predictive distributions can potentially serve as useful decision aids which should be used with caution and take into account the type of distribution and the expertise of the human.
  arXiv  Detail & Related papers  (2020-11-12T02:23:53Z)
• Counterfactual Predictions under Runtime Confounding [74.90756694584839]
  We study the counterfactual prediction task in the setting where all relevant factors are captured in the historical data. We propose a doubly-robust procedure for learning counterfactual prediction models in this setting.
  arXiv  Detail & Related papers  (2020-06-30T15:49:05Z)
• Individual Calibration with Randomized Forecasting [116.2086707626651]
  We show that calibration for individual samples is possible in the regression setup if the predictions are randomized. We design a training objective to enforce individual calibration and use it to train randomized regression functions.
  arXiv  Detail & Related papers  (2020-06-18T05:53:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.