Related papers: Calibration Error for Decision Making

Calibration Error for Decision Making

URL: http://arxiv.org/abs/2404.13503v5
Date: Thu, 10 Oct 2024 22:15:26 GMT
Title: Calibration Error for Decision Making
Authors: Lunjia Hu, Yifan Wu,
Abstract summary: We propose a decision-theoretic calibration error, the Decision Loss (CDL), defined as the maximum improvement in decision payoff obtained by the predictions. We show separations between CDL and existing calibration error metrics, including the most well-studied metric Expected Error (ECE) Our main technical contribution is a new efficient algorithm for online calibration that achieves near-optimal $O(fraclog TsqrtT)$ expected CDL.
Score: 15.793486463552144
License: http://creativecommons.org/licenses/by/4.0/
Abstract: Calibration allows predictions to be reliably interpreted as probabilities by decision makers. We propose a decision-theoretic calibration error, the Calibration Decision Loss (CDL), defined as the maximum improvement in decision payoff obtained by calibrating the predictions, where the maximum is over all payoff-bounded decision tasks. Vanishing CDL guarantees the payoff loss from miscalibration vanishes simultaneously for all downstream decision tasks. We show separations between CDL and existing calibration error metrics, including the most well-studied metric Expected Calibration Error (ECE). Our main technical contribution is a new efficient algorithm for online calibration that achieves near-optimal $O(\frac{\log T}{\sqrt{T}})$ expected CDL, bypassing the $\Omega(T^{-0.472})$ lower bound for ECE by Qiao and Valiant (2021).

Related papers

Smooth Calibration and Decision Making [11.51844809748468]
We show that post-processing an online predictor with $eps$ to calibration achieves $O(sqrtepsilon)$ ECE and CDL. The optimal bound is non-optimal compared with existing online calibration algorithms.
arXiv Detail & Related papers (2025-04-22T04:55:41Z)
Truthfulness of Decision-Theoretic Calibration Measures [5.414308305392762]
We introduce a new calibration measure termed subsampled step calibration, $mathsfStepCEtextsfsub$, that is both decision-theoretic and truthful. In particular, on any product distribution, $mathsfStepCEtextsfsub$ is truthful up to an $O(1)$ factor whereas prior decision-theoretic calibration measures suffer from an $e-Omega(T)$-$Omega(sqrtT)$ truthfulness gap.
arXiv Detail & Related papers (2025-03-04T08:20:10Z)
Orthogonal Causal Calibration [55.28164682911196]
We prove generic upper bounds on the calibration error of any causal parameter estimate $theta$ with respect to any loss $ell$. We use our bound to analyze the convergence of two sample splitting algorithms for causal calibration.
arXiv Detail & Related papers (2024-06-04T03:35:25Z)
Towards Certification of Uncertainty Calibration under Adversarial Attacks [96.48317453951418]
We show that attacks can significantly harm calibration, and thus propose certified calibration as worst-case bounds on calibration under adversarial perturbations. We propose novel calibration attacks and demonstrate how they can improve model calibration through textitadversarial calibration training
arXiv Detail & Related papers (2024-05-22T18:52:09Z)
Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
arXiv Detail & Related papers (2023-10-31T06:19:40Z)
Calibration Error Estimation Using Fuzzy Binning [0.0]
We propose a Fuzzy Error metric (FCE) that utilizes a fuzzy binning approach to calculate calibration error. Our results show that FCE offers better calibration error estimation, especially in multi-class settings.
arXiv Detail & Related papers (2023-04-30T18:06:14Z)
Calibrating Predictions to Decisions: A Novel Approach to Multi-Class Calibration [118.26862029820447]
We introduce a new notion -- emphdecision calibration -- that requires the predicted distribution and true distribution to be indistinguishable'' to a set of downstream decision-makers. Decision calibration improves decision-making on skin lesions and ImageNet classification with modern neural network.
arXiv Detail & Related papers (2021-07-12T20:17:28Z)
Localized Calibration: Metrics and Recalibration [133.07044916594361]
We propose a fine-grained calibration metric that spans the gap between fully global and fully individualized calibration. We then introduce a localized recalibration method, LoRe, that improves the LCE better than existing recalibration methods.
arXiv Detail & Related papers (2021-02-22T07:22:12Z)
Transferable Calibration with Lower Bias and Variance in Domain Adaptation [139.4332115349543]
Domain Adaptation (DA) enables transferring a learning machine from a labeled source domain to an unlabeled target one. How to estimate the predictive uncertainty of DA models is vital for decision-making in safety-critical scenarios. TransCal can be easily applied to recalibrate existing DA methods.
arXiv Detail & Related papers (2020-07-16T11:09:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.