Calibration of Neural Networks using Splines
- URL: http://arxiv.org/abs/2006.12800v2
- Date: Wed, 29 Dec 2021 17:58:53 GMT
- Title: Calibration of Neural Networks using Splines
- Authors: Kartik Gupta, Amir Rahimi, Thalaiyasingam Ajanthan, Thomas Mensink,
Cristian Sminchisescu, Richard Hartley
- Abstract summary: Measuring calibration error amounts to comparing two empirical distributions.
We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test.
Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
- Score: 51.42640515410253
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Calibrating neural networks is of utmost importance when employing them in
safety-critical applications where the downstream decision making depends on
the predicted probabilities. Measuring calibration error amounts to comparing
two empirical distributions. In this work, we introduce a binning-free
calibration measure inspired by the classical Kolmogorov-Smirnov (KS)
statistical test in which the main idea is to compare the respective cumulative
probability distributions. From this, by approximating the empirical cumulative
distribution using a differentiable function via splines, we obtain a
recalibration function, which maps the network outputs to actual (calibrated)
class assignment probabilities. The spine-fitting is performed using a held-out
calibration set and the obtained recalibration function is evaluated on an
unseen test set. We tested our method against existing calibration approaches
on various image classification datasets and our spline-based recalibration
approach consistently outperforms existing methods on KS error as well as other
commonly used calibration measures. Our Code is available at
https://github.com/kartikgupta-at-anu/spline-calibration.
Related papers
- Optimizing Estimators of Squared Calibration Errors in Classification [2.3020018305241337]
We propose a mean-squared error-based risk that enables the comparison and optimization of estimators of squared calibration errors.
Our approach advocates for a training-validation-testing pipeline when estimating a calibration error.
arXiv Detail & Related papers (2024-10-09T15:58:06Z) - ForeCal: Random Forest-based Calibration for DNNs [0.0]
We propose ForeCal, a novel post-hoc calibration algorithm based on Random forests.
ForeCal exploits two unique properties of Random forests: the ability to enforce weak monotonicity and range-preservation.
We show that ForeCal outperforms existing methods in terms of Expected Error(ECE) with minimal impact on the discriminative power of the base as measured by AUC.
arXiv Detail & Related papers (2024-09-04T04:56:41Z) - Consistent and Asymptotically Unbiased Estimation of Proper Calibration
Errors [23.819464242327257]
We propose a method that allows consistent estimation of all proper calibration errors and refinement terms.
We prove the relation between refinement and f-divergences, which implies information monotonicity in neural networks.
Our experiments validate the claimed properties of the proposed estimator and suggest that the selection of a post-hoc calibration method should be determined by the particular calibration error of interest.
arXiv Detail & Related papers (2023-12-14T01:20:08Z) - Calibration by Distribution Matching: Trainable Kernel Calibration
Metrics [56.629245030893685]
We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression.
These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization.
We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
arXiv Detail & Related papers (2023-10-31T06:19:40Z) - Calibration of Neural Networks [77.34726150561087]
This paper presents a survey of confidence calibration problems in the context of neural networks.
We analyze problem statement, calibration definitions, and different approaches to evaluation.
Empirical experiments cover various datasets and models, comparing calibration methods according to different criteria.
arXiv Detail & Related papers (2023-03-19T20:27:51Z) - Parametric and Multivariate Uncertainty Calibration for Regression and
Object Detection [4.630093015127541]
We show that common detection models overestimate the spatial uncertainty in comparison to the observed error.
Our experiments show that the simple Isotonic Regression recalibration method is sufficient to achieve a good calibrated uncertainty.
In contrast, if normal distributions are required for subsequent processes, our GP-Normal recalibration method yields the best results.
arXiv Detail & Related papers (2022-07-04T08:00:20Z) - Localized Calibration: Metrics and Recalibration [133.07044916594361]
We propose a fine-grained calibration metric that spans the gap between fully global and fully individualized calibration.
We then introduce a localized recalibration method, LoRe, that improves the LCE better than existing recalibration methods.
arXiv Detail & Related papers (2021-02-22T07:22:12Z) - Unsupervised Calibration under Covariate Shift [92.02278658443166]
We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it.
We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
arXiv Detail & Related papers (2020-06-29T21:50:07Z) - Distribution-free binary classification: prediction sets, confidence
intervals and calibration [106.50279469344937]
We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting.
We derive confidence intervals for binned probabilities for both fixed-width and uniform-mass binning.
As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration.
arXiv Detail & Related papers (2020-06-18T14:17:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.