Improving Calibration by Relating Focal Loss, Temperature Scaling, and Properness

• URL: http://arxiv.org/abs/2408.11598v1
• Date: Wed, 21 Aug 2024 13:10:44 GMT
• Title: Improving Calibration by Relating Focal Loss, Temperature Scaling, and Properness
• Authors: Viacheslav Komisarenko, Meelis Kull,
• Abstract summary: Cross-entropy incentivizes classifiers to produce class probabilities that are well-calibrated on the training data. We show that focal loss can be decomposed into a confidence-raising transformation and a proper loss. We propose focal temperature scaling - a new post-hoc calibration method combining focal calibration and temperature scaling.
• Score: 1.9055459597116435
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Proper losses such as cross-entropy incentivize classifiers to produce class probabilities that are well-calibrated on the training data. Due to the generalization gap, these classifiers tend to become overconfident on the test data, mandating calibration methods such as temperature scaling. The focal loss is not proper, but training with it has been shown to often result in classifiers that are better calibrated on test data. Our first contribution is a simple explanation about why focal loss training often leads to better calibration than cross-entropy training. For this, we prove that focal loss can be decomposed into a confidence-raising transformation and a proper loss. This is why focal loss pushes the model to provide under-confident predictions on the training data, resulting in being better calibrated on the test data, due to the generalization gap. Secondly, we reveal a strong connection between temperature scaling and focal loss through its confidence-raising transformation, which we refer to as the focal calibration map. Thirdly, we propose focal temperature scaling - a new post-hoc calibration method combining focal calibration and temperature scaling. Our experiments on three image classification datasets demonstrate that focal temperature scaling outperforms standard temperature scaling.

Related papers

• Calibrating Language Models with Adaptive Temperature Scaling [58.056023173579625]
  We introduce Adaptive Temperature Scaling (ATS), a post-hoc calibration method that predicts a temperature scaling parameter for each token prediction. ATS improves calibration by over 10-50% across three downstream natural language evaluation benchmarks compared to prior calibration methods.
  arXiv  Detail & Related papers  (2024-09-29T22:54:31Z)
• On the Limitations of Temperature Scaling for Distributions with Overlaps [8.486166869140929]
  We show that for empirical risk minimizers for a general set of distributions, the performance of temperature scaling degrades with the amount of overlap between classes. We prove that optimizing a modified form of the empirical risk induced by the Mixup data augmentation technique can in fact lead to reasonably good calibration performance.
  arXiv  Detail & Related papers  (2023-06-01T14:35:28Z)
• AdaFocal: Calibration-aware Adaptive Focal Loss [8.998525155518836]
  Training with focal loss leads to better calibration than cross-entropy. We propose a calibration-aware adaptive focal loss called AdaFocal.
  arXiv  Detail & Related papers  (2022-11-21T20:19:24Z)
• A Close Look into the Calibration of Pre-trained Language Models [56.998539510508515]
  Pre-trained language models (PLMs) may fail in giving reliable estimates of their predictive uncertainty. We study the dynamic change in PLMs' calibration performance in training. We extend two recently proposed learnable methods that directly collect data to train models to have reasonable confidence estimations.
  arXiv  Detail & Related papers  (2022-10-31T21:31:07Z)
• Adaptive Temperature Scaling for Robust Calibration of Deep Neural Networks [0.7219077740523682]
  We focus on the task of confidence scaling, specifically on post-hoc methods that generalize Temperature Scaling. We show that when there is plenty of data complex models like neural networks yield better performance, but are prone to fail when the amount of data is limited. We propose Entropy-based Temperature Scaling, a simple method that scales the confidence of a prediction according to its entropy.
  arXiv  Detail & Related papers  (2022-07-31T16:20:06Z)
• Sample-dependent Adaptive Temperature Scaling for Improved Calibration [95.7477042886242]
  Post-hoc approach to compensate for neural networks being wrong is to perform temperature scaling. We propose to predict a different temperature value for each input, allowing us to adjust the mismatch between confidence and accuracy. We test our method on the ResNet50 and WideResNet28-10 architectures using the CIFAR10/100 and Tiny-ImageNet datasets.
  arXiv  Detail & Related papers  (2022-07-13T14:13:49Z)
• Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibration [57.568461777747515]
  We introduce a novel calibration method, Parametrized Temperature Scaling (PTS) We demonstrate that the performance of accuracy-preserving state-of-the-art post-hoc calibrators is limited by their intrinsic expressive power. We show with extensive experiments that our novel accuracy-preserving approach consistently outperforms existing algorithms across a large number of model architectures, datasets and metrics.
  arXiv  Detail & Related papers  (2021-02-24T10:18:30Z)
• Uncertainty Quantification and Deep Ensembles [79.4957965474334]
  We show that deep-ensembles do not necessarily lead to improved calibration properties. We show that standard ensembling methods, when used in conjunction with modern techniques such as mixup regularization, can lead to less calibrated models. This text examines the interplay between three of the most simple and commonly used approaches to leverage deep learning when data is scarce.
  arXiv  Detail & Related papers  (2020-07-17T07:32:24Z)
• Calibrating Deep Neural Networks using Focal Loss [77.92765139898906]
  Miscalibration is a mismatch between a model's confidence and its correctness. We show that focal loss allows us to learn models that are already very well calibrated. We show that our approach achieves state-of-the-art calibration without compromising on accuracy in almost all cases.
  arXiv  Detail & Related papers  (2020-02-21T17:35:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.