Measuring multi-calibration

• URL: http://arxiv.org/abs/2506.11251v1
• Date: Thu, 12 Jun 2025 19:48:10 GMT
• Title: Measuring multi-calibration
• Abstract summary: When predicted probabilities are perfectly calibrated simultaneously across several subpopulations, the predictions are known as "perfectly multi-calibrated"<n>A recently proposed metric for calibration, based on the classical Kuiper statistic, is a natural basis for a new metric of multi-calibration.
• Score: 9.514997419494993
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: A suitable scalar metric can help measure multi-calibration, defined as follows. When the expected values of observed responses are equal to corresponding predicted probabilities, the probabilistic predictions are known as "perfectly calibrated." When the predicted probabilities are perfectly calibrated simultaneously across several subpopulations, the probabilistic predictions are known as "perfectly multi-calibrated." In practice, predicted probabilities are seldom perfectly multi-calibrated, so a statistic measuring the distance from perfect multi-calibration is informative. A recently proposed metric for calibration, based on the classical Kuiper statistic, is a natural basis for a new metric of multi-calibration and avoids well-known problems of metrics based on binning or kernel density estimation. The newly proposed metric weights the contributions of different subpopulations in proportion to their signal-to-noise ratios; data analyses' ablations demonstrate that the metric becomes noisy when omitting the signal-to-noise ratios from the metric. Numerical examples on benchmark data sets illustrate the new metric.

Related papers

• From token probabilities to calibrated confidence: An empirical study of mathematical question answering [1.0032961794537367]
  Prior work has shown that token probabilities are often overconfident.<n>We compare single-pass estimators, which reuse token probabilities from the original generation, with multi-pass estimators.<n>We find that aggregating token probabilities over the full sequence captures small but consistent differences between correct and incorrect generations.
  arXiv  Detail & Related papers  (2026-08-08T00:00:19Z)
• Finite-Shot Sensitivity for Moment Estimation in Quantum Metrology [3.5426941871022373]
  We develop a finite-measurement theory for method-of-moments estimation.<n>We identify a general density-matrix condition under which the full $1/2$ correction vanishes.<n>The resulting thresholds quantify how many measurements are needed before the sensitivity of a moment- protocol protocol is operationally visible.
  arXiv  Detail & Related papers  (2026-06-24T15:01:01Z)
• Scalable Utility-Aware Multiclass Calibration [53.28176049547449]
  Utility calibration is a general framework that measures the calibration error relative to a specific utility function.<n>We demonstrate how this framework can unify and re-interpret several existing calibration metrics.
  arXiv  Detail & Related papers  (2025-10-29T12:32:14Z)
• Learning Survival Distributions with the Asymmetric Laplace Distribution [16.401141867387324]
  We propose a parametric survival analysis method based on the Asymmetric Laplace Distribution (ALD)<n>This distribution allows for closed-form calculation of popular event summaries such as mean, median, mode, variation, and quantiles.<n>We show that the proposed method outperforms parametric and nonparametric approaches in terms of accuracy, discrimination and calibration.
  arXiv  Detail & Related papers  (2025-05-06T17:34:41Z)
• Semiparametric conformal prediction [79.6147286161434]
  We construct a conformal prediction set accounting for the joint correlation structure of the vector-valued non-conformity scores.<n>We flexibly estimate the joint cumulative distribution function (CDF) of the scores.<n>Our method yields desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate.<n>We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• Distribution-Free Model-Agnostic Regression Calibration via Nonparametric Methods [9.662269016653296]
  We consider an individual calibration objective for characterizing the quantiles of the prediction model. Existing methods have been largely and lack of statistical guarantee in terms of individual calibration. We propose simple nonparametric calibration methods that are agnostic of the underlying prediction model.
  arXiv  Detail & Related papers  (2023-05-20T21:31:51Z)
• Multi-Fidelity Covariance Estimation in the Log-Euclidean Geometry [0.0]
  We introduce a multi-fidelity estimator of covariance matrices that employs the log-Euclidean geometry of the symmetric positive-definite manifold. We develop an optimal sample allocation scheme that minimizes the mean-squared error of the estimator given a fixed budget. Evaluations of our approach using data from physical applications demonstrate more accurate metric learning and speedups of more than one order of magnitude compared to benchmarks.
  arXiv  Detail & Related papers  (2023-01-31T16:33:46Z)
• Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection [4.630093015127541]
  We show that common detection models overestimate the spatial uncertainty in comparison to the observed error. Our experiments show that the simple Isotonic Regression recalibration method is sufficient to achieve a good calibrated uncertainty. In contrast, if normal distributions are required for subsequent processes, our GP-Normal recalibration method yields the best results.
  arXiv  Detail & Related papers  (2022-07-04T08:00:20Z)
• Metrics of calibration for probabilistic predictions [0.0]
  "Reliability diagrams" help detect and diagnose statistically significant discrepancies -- so-called "miscalibration" The canonical reliability diagrams histogram the observed and expected values of the predictions. But, which widths of bins or kernels are best? Slope is easy to perceive with quantitative precision, even when the constant offsets of the secant lines are irrelevant.
  arXiv  Detail & Related papers  (2022-05-19T16:38:24Z)
• Can a single neuron learn predictive uncertainty? [0.0]
  We introduce a novel non-parametric quantile estimation method for continuous random variables based on the simplest neural network architecture with one degree of freedom: a single neuron. In real-world applications, the method can be used to quantify predictive uncertainty under the split conformal prediction setting.
  arXiv  Detail & Related papers  (2021-06-07T15:12:47Z)
• A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methods [60.04142561088524]
  We find that the confidence intervals are rather wide, demonstrating high uncertainty in how reliable automatic metrics truly are. Although many metrics fail to show statistical improvements over ROUGE, two recent works, QAEval and BERTScore, do in some evaluation settings.
  arXiv  Detail & Related papers  (2021-03-31T18:28:14Z)
• Localized Calibration: Metrics and Recalibration [133.07044916594361]
  We propose a fine-grained calibration metric that spans the gap between fully global and fully individualized calibration. We then introduce a localized recalibration method, LoRe, that improves the LCE better than existing recalibration methods.
  arXiv  Detail & Related papers  (2021-02-22T07:22:12Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.