Multi-Head Multi-Loss Model Calibration

• URL: http://arxiv.org/abs/2303.01099v1
• Date: Thu, 2 Mar 2023 09:32:32 GMT
• Title: Multi-Head Multi-Loss Model Calibration
• Authors: Adrian Galdran, Johan Verjans, Gustavo Carneiro, Miguel A. Gonz\'alez Ballester
• Abstract summary: We introduce a form of simplified ensembling that bypasses the costly training and inference of deep ensembles. Specifically, each head is trained to minimize a weighted Cross-Entropy loss, but the weights are different among the different branches. We show that the resulting averaged predictions can achieve excellent calibration without sacrificing accuracy in two challenging datasets.
• Score: 13.841172927454204
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Delivering meaningful uncertainty estimates is essential for a successful deployment of machine learning models in the clinical practice. A central aspect of uncertainty quantification is the ability of a model to return predictions that are well-aligned with the actual probability of the model being correct, also known as model calibration. Although many methods have been proposed to improve calibration, no technique can match the simple, but expensive approach of training an ensemble of deep neural networks. In this paper we introduce a form of simplified ensembling that bypasses the costly training and inference of deep ensembles, yet it keeps its calibration capabilities. The idea is to replace the common linear classifier at the end of a network by a set of heads that are supervised with different loss functions to enforce diversity on their predictions. Specifically, each head is trained to minimize a weighted Cross-Entropy loss, but the weights are different among the different branches. We show that the resulting averaged predictions can achieve excellent calibration without sacrificing accuracy in two challenging datasets for histopathological and endoscopic image classification. Our experiments indicate that Multi-Head Multi-Loss classifiers are inherently well-calibrated, outperforming other recent calibration techniques and even challenging Deep Ensembles' performance. Code to reproduce our experiments can be found at \url{https://github.com/agaldran/mhml_calibration} .

Related papers

• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• Calibration of Neural Networks [77.34726150561087]
  This paper presents a survey of confidence calibration problems in the context of neural networks. We analyze problem statement, calibration definitions, and different approaches to evaluation. Empirical experiments cover various datasets and models, comparing calibration methods according to different criteria.
  arXiv  Detail & Related papers  (2023-03-19T20:27:51Z)
• On Calibrating Semantic Segmentation Models: Analyses and An Algorithm [51.85289816613351]
  We study the problem of semantic segmentation calibration. Model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. We propose a simple, unifying, and effective approach, namely selective scaling.
  arXiv  Detail & Related papers  (2022-12-22T22:05:16Z)
• Class Adaptive Network Calibration [19.80805957502909]
  We propose Class Adaptive Label Smoothing (CALS) for calibrating deep networks. Our method builds on a general Augmented Lagrangian approach, a well-established technique in constrained optimization.
  arXiv  Detail & Related papers  (2022-11-28T06:05:31Z)
• Class-wise and reduced calibration methods [0.0]
  We show how a reduced calibration method transforms the original problem into a simpler one. Second, we propose class-wise calibration methods, based on building on a phenomenon called neural collapse. Applying the two methods together results in class-wise reduced calibration algorithms, which are powerful tools for reducing the prediction and per-class calibration errors.
  arXiv  Detail & Related papers  (2022-10-07T17:13:17Z)
• Calibrating Ensembles for Scalable Uncertainty Quantification in Deep Learning-based Medical Segmentation [0.42008820076301906]
  Uncertainty quantification in automated image analysis is highly desired in many applications. Current uncertainty quantification approaches do not scale well in high-dimensional real-world problems. We propose a scalable and intuitive framework to calibrate ensembles of deep learning models to produce uncertainty quantification measurements.
  arXiv  Detail & Related papers  (2022-09-20T09:09:48Z)
• Learning to Learn to Demodulate with Uncertainty Quantification via Bayesian Meta-Learning [59.014197664747165]
  We introduce the use of Bayesian meta-learning via variational inference for the purpose of obtaining well-calibrated few-pilot demodulators. The resulting Bayesian ensembles offer better calibrated soft decisions, at the computational cost of running multiple instances of the neural network for demodulation.
  arXiv  Detail & Related papers  (2021-08-02T11:07:46Z)
• Improved Trainable Calibration Method for Neural Networks on Medical Imaging Classification [17.941506832422192]
  Empirically, neural networks are often miscalibrated and overconfident in their predictions. We propose a novel calibration approach that maintains the overall classification accuracy while significantly improving model calibration.
  arXiv  Detail & Related papers  (2020-09-09T01:25:53Z)
• Regularizing Class-wise Predictions via Self-knowledge Distillation [80.76254453115766]
  We propose a new regularization method that penalizes the predictive distribution between similar samples. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve the generalization ability.
  arXiv  Detail & Related papers  (2020-03-31T06:03:51Z)
• Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks [54.23874144090228]
  A common approach is to learn a post-hoc calibration function that transforms the output of the original network into calibrated confidence scores. Previous post-hoc calibration techniques work only with simple calibration functions. We propose a new neural network architecture that represents a class of intra order-preserving functions.
  arXiv  Detail & Related papers  (2020-03-15T12:57:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.