Calibration of Neural Networks

• URL: http://arxiv.org/abs/2303.10761v1
• Date: Sun, 19 Mar 2023 20:27:51 GMT
• Title: Calibration of Neural Networks
• Authors: Ruslan Vasilev, Alexander D'yakonov
• Abstract summary: This paper presents a survey of confidence calibration problems in the context of neural networks. We analyze problem statement, calibration definitions, and different approaches to evaluation. Empirical experiments cover various datasets and models, comparing calibration methods according to different criteria.
• Score: 77.34726150561087
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural networks solving real-world problems are often required not only to make accurate predictions but also to provide a confidence level in the forecast. The calibration of a model indicates how close the estimated confidence is to the true probability. This paper presents a survey of confidence calibration problems in the context of neural networks and provides an empirical comparison of calibration methods. We analyze problem statement, calibration definitions, and different approaches to evaluation: visualizations and scalar measures that estimate whether the model is well-calibrated. We review modern calibration techniques: based on post-processing or requiring changes in training. Empirical experiments cover various datasets and models, comparing calibration methods according to different criteria.

Related papers

• Probabilistic Calibration by Design for Neural Network Regression [2.3020018305241337]
  We introduce a novel end-to-end model training procedure called Quantile Recalibration Training. We demonstrate the performance of our method in a large-scale experiment involving 57 regression datasets.
  arXiv  Detail & Related papers  (2024-03-18T17:04:33Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration [1.192436948211501]
  Proper confidence calibration of deep neural networks is essential for reliable predictions in safety-critical tasks. Miscalibration can lead to model over-confidence and/or under-confidence. We introduce a novel metric, a miscalibration score, to identify the overall and class-wise calibration status. We use the class-wise miscalibration score as a proxy to design a calibration technique that can tackle both over and under-confidence.
  arXiv  Detail & Related papers  (2023-08-06T17:59:14Z)
• Calibration in Deep Learning: A Survey of the State-of-the-Art [7.6087138685470945]
  Calibrating deep neural models plays an important role in building reliable, robust AI systems in safety-critical applications. Recent work has shown that modern neural networks that possess high predictive capability are poorly calibrated and produce unreliable model predictions.
  arXiv  Detail & Related papers  (2023-08-02T15:28:10Z)
• On Calibrating Semantic Segmentation Models: Analyses and An Algorithm [51.85289816613351]
  We study the problem of semantic segmentation calibration. Model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. We propose a simple, unifying, and effective approach, namely selective scaling.
  arXiv  Detail & Related papers  (2022-12-22T22:05:16Z)
• Meta-Calibration: Learning of Model Calibration Using Differentiable Expected Calibration Error [46.12703434199988]
  We introduce a new differentiable surrogate for expected calibration error (DECE) that allows calibration quality to be directly optimised. We also propose a meta-learning framework that uses DECE to optimise for validation set calibration.
  arXiv  Detail & Related papers  (2021-06-17T15:47:50Z)
• Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibration [57.568461777747515]
  We introduce a novel calibration method, Parametrized Temperature Scaling (PTS) We demonstrate that the performance of accuracy-preserving state-of-the-art post-hoc calibrators is limited by their intrinsic expressive power. We show with extensive experiments that our novel accuracy-preserving approach consistently outperforms existing algorithms across a large number of model architectures, datasets and metrics.
  arXiv  Detail & Related papers  (2021-02-24T10:18:30Z)
• Unsupervised Calibration under Covariate Shift [92.02278658443166]
  We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
  arXiv  Detail & Related papers  (2020-06-29T21:50:07Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.