Hierarchies of Calibration: Classification meets Regression

• URL: http://arxiv.org/abs/2606.03245v1
• Date: Tue, 02 Jun 2026 07:05:20 GMT
• Title: Hierarchies of Calibration: Classification meets Regression
• Abstract summary: Concepts of calibration formalize the compatibility between probabilistic predictions and the respective outcomes.<n>In this paper, we review, extend, and bridge notions of calibration that have been proposed for classification and regression tasks.
• Score: 4.90063885770646
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Concepts of calibration formalize the compatibility between probabilistic predictions and the respective outcomes. In a nutshell, the outcomes ought to be indistinguishable from random draws from the predictive distributions. In this paper, we review, extend, and bridge notions of calibration that have been proposed for classification and regression tasks. Particular emphasis is given to hierarchical relations between the various notions, as they apply to general real-valued data, continuous outcomes, count data, nominal classes, and binary outcomes. To highlight a number of contributions, we introduce the notion of modal calibration for nominal outcomes, we distinguish full, partial, and average calibration in this setting, and we show that double probability integral transform (PIT) calibration is logically independent of previously proposed concepts of calibration for discrete outcomes. Furthermore, we generalize extant results on concepts of calibration that are expressed in terms of properties or functionals of the predictive distributions, such as means, quantiles, or event probabilities. Throughout the paper, we illustrate the concepts and their hierarchical relations in worked examples, and we provide algorithmic tools that support the construction of instructive examples and counterexamples.

Related papers

• Aggregation of Statistical Evidence under Exchangeability [22.375838048747294]
  We study aggregation of statistical evidence under unknown and potentially complex dependence.<n>We develop a finite-sample power and adaptivity theory for this framework.
  arXiv  Detail & Related papers  (2026-07-17T10:32:05Z)
• On Optimal Data Splitting for Split Conformal Prediction [0.7205546587515871]
  We develop a theoretical framework for optimal data splitting in split conformal prediction.<n>We show how the general results specialize to several commonly used regression settings.<n>Our analysis clarifies how model-related features govern the optimal allocation of samples between training and calibration.
  arXiv  Detail & Related papers  (2026-06-30T12:49:44Z)
• Partial Transportability for Domain Generalization [56.37032680901525]
  Building on the theory of partial identification and transportability, this paper introduces new results for bounding the value of a functional of the target distribution.<n>Our contribution is to provide the first general estimation technique for transportability problems.<n>We propose a gradient-based optimization scheme for making scalable inferences in practice.
  arXiv  Detail & Related papers  (2025-03-30T22:06:37Z)
• Calibrated Probabilistic Forecasts for Arbitrary Sequences [58.54729945445505]
  Real-world data streams can change unpredictably due to distribution shifts, feedback loops and adversarial actors.<n>We present a forecasting framework ensuring valid uncertainty estimates regardless of how data evolves.
  arXiv  Detail & Related papers  (2024-09-27T21:46:42Z)
• Multiply-Robust Causal Change Attribution [15.501106533308798]
  We develop a new estimation strategy that combines regression and re-weighting methods to quantify the contribution of each causal mechanism. Our method demonstrates excellent performance in Monte Carlo simulations, and we show its usefulness in an empirical application.
  arXiv  Detail & Related papers  (2024-04-12T22:57:01Z)
• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• Learning Invariant Representations under General Interventions on the Response [2.725698729450241]
  We focus on linear structural causal models (SCMs) and introduce invariant matching property (IMP) We analyze the generalization errors of our method under both the discrete and continuous environment settings.
  arXiv  Detail & Related papers  (2022-08-22T03:09:17Z)
• Predicting Out-of-Domain Generalization with Neighborhood Invariance [59.05399533508682]
  We propose a measure of a classifier's output invariance in a local transformation neighborhood. Our measure is simple to calculate, does not depend on the test point's true label, and can be applied even in out-of-domain (OOD) settings. In experiments on benchmarks in image classification, sentiment analysis, and natural language inference, we demonstrate a strong and robust correlation between our measure and actual OOD generalization.
  arXiv  Detail & Related papers  (2022-07-05T14:55:16Z)
• An Invariant Matching Property for Distribution Generalization under Intervened Response [19.786769414376323]
  We show a novel form of invariance by incorporating the estimates of certain features as additional predictors. We provide an explicit characterization of the linear matching and present our simulation results under various intervention settings.
  arXiv  Detail & Related papers  (2022-05-18T18:25:21Z)
• Distributional Gradient Boosting Machines [77.34726150561087]
  Our framework is based on XGBoost and LightGBM. We show that our framework achieves state-of-the-art forecast accuracy.
  arXiv  Detail & Related papers  (2022-04-02T06:32:19Z)
• Deconfounding Scores: Feature Representations for Causal Effect Estimation with Weak Overlap [140.98628848491146]
  We introduce deconfounding scores, which induce better overlap without biasing the target of estimation. We show that deconfounding scores satisfy a zero-covariance condition that is identifiable in observed data. In particular, we show that this technique could be an attractive alternative to standard regularizations.
  arXiv  Detail & Related papers  (2021-04-12T18:50:11Z)
• Should Ensemble Members Be Calibrated? [16.331175260764]
  Modern deep neural networks are often observed to be poorly calibrated. Deep learning approaches make use of large numbers of model parameters. This paper explores the application of calibration schemes to deep ensembles.
  arXiv  Detail & Related papers  (2021-01-13T23:59:00Z)
• From interpretability to inference: an estimation framework for universal approximators [0.0]
  We present a novel framework for estimation and inference with the broad class of universal approximators.<n>Inference relies on analyzing the bias and variance properties of individual Shapley components.
  arXiv  Detail & Related papers  (2019-03-11T10:37:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.