Classifier Calibration: How to assess and improve predicted class probabilities: a survey

• URL: http://arxiv.org/abs/2112.10327v1
• Date: Mon, 20 Dec 2021 03:50:55 GMT
• Title: Classifier Calibration: How to assess and improve predicted class probabilities: a survey
• Authors: Telmo Silva Filho, Hao Song, Miquel Perello-Nieto, Raul Santos-Rodriguez, Meelis Kull, Peter Flach
• Abstract summary: A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change.
• Score: 10.587567878098444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This paper provides both an introduction to and a detailed overview of the principles and practice of classifier calibration. A well-calibrated classifier correctly quantifies the level of uncertainty or confidence associated with its instance-wise predictions. This is essential for critical applications, optimal decision making, cost-sensitive classification, and for some types of context change. Calibration research has a rich history which predates the birth of machine learning as an academic field by decades. However, a recent increase in the interest on calibration has led to new methods and the extension from binary to the multiclass setting. The space of options and issues to consider is large, and navigating it requires the right set of concepts and tools. We provide both introductory material and up-to-date technical details of the main concepts and methods, including proper scoring rules and other evaluation metrics, visualisation approaches, a comprehensive account of post-hoc calibration methods for binary and multiclass classification, and several advanced topics.

Related papers

• Bayesian Test-Time Adaptation for Vision-Language Models [51.93247610195295]
  Test-time adaptation with pre-trained vision-language models, such as CLIP, aims to adapt the model to new, potentially out-of-distribution test data. We propose a novel approach, textbfBayesian textbfClass textbfAdaptation (BCA), which in addition to continuously updating class embeddings to adapt likelihood, also uses the posterior of incoming samples to continuously update the prior for each class embedding.
  arXiv  Detail & Related papers  (2025-03-12T10:42:11Z)
• Rethinking Early Stopping: Refine, Then Calibrate [49.966899634962374]
  We present a novel variational formulation of the calibration-refinement decomposition.<n>We provide theoretical and empirical evidence that calibration and refinement errors are not minimized simultaneously during training.
  arXiv  Detail & Related papers  (2025-01-31T15:03:54Z)
• Anomaly Detection using Ensemble Classification and Evidence Theory [62.997667081978825]
  We present a novel approach for novel detection using ensemble classification and evidence theory. A pool selection strategy is presented to build a solid ensemble classifier. We use uncertainty for the anomaly detection approach.
  arXiv  Detail & Related papers  (2022-12-23T00:50:41Z)
• On Calibrating Semantic Segmentation Models: Analyses and An Algorithm [51.85289816613351]
  We study the problem of semantic segmentation calibration. Model capacity, crop size, multi-scale testing, and prediction correctness have impact on calibration. We propose a simple, unifying, and effective approach, namely selective scaling.
  arXiv  Detail & Related papers  (2022-12-22T22:05:16Z)
• Class Adaptive Network Calibration [19.80805957502909]
  We propose Class Adaptive Label Smoothing (CALS) for calibrating deep networks. Our method builds on a general Augmented Lagrangian approach, a well-established technique in constrained optimization.
  arXiv  Detail & Related papers  (2022-11-28T06:05:31Z)
• Learning Acceptance Regions for Many Classes with Anomaly Detection [19.269724165953274]
  Many existing set-valued classification methods do not consider the possibility that a new class that never appeared in the training data appears in the test data. We propose a Generalized Prediction Set (GPS) approach to estimate the acceptance regions while considering the possibility of a new class in the test data. Unlike previous methods, the proposed method achieves a good balance between accuracy, efficiency, and anomaly detection rate.
  arXiv  Detail & Related papers  (2022-09-20T19:40:33Z)
• Towards Diverse Evaluation of Class Incremental Learning: A Representation Learning Perspective [67.45111837188685]
  Class incremental learning (CIL) algorithms aim to continually learn new object classes from incrementally arriving data. We experimentally analyze neural network models trained by CIL algorithms using various evaluation protocols in representation learning.
  arXiv  Detail & Related papers  (2022-06-16T11:44:11Z)
• Class-Incremental Learning with Strong Pre-trained Models [97.84755144148535]
  Class-incremental learning (CIL) has been widely studied under the setting of starting from a small number of classes (base classes) We explore an understudied real-world setting of CIL that starts with a strong model pre-trained on a large number of base classes. Our proposed method is robust and generalizes to all analyzed CIL settings.
  arXiv  Detail & Related papers  (2022-04-07T17:58:07Z)
• Theoretical Insights Into Multiclass Classification: A High-dimensional Asymptotic View [82.80085730891126]
  We provide the first modernally precise analysis of linear multiclass classification. Our analysis reveals that the classification accuracy is highly distribution-dependent. The insights gained may pave the way for a precise understanding of other classification algorithms.
  arXiv  Detail & Related papers  (2020-11-16T05:17:29Z)
• Unsupervised Calibration under Covariate Shift [92.02278658443166]
  We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
  arXiv  Detail & Related papers  (2020-06-29T21:50:07Z)
• Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning [8.780958735684958]
  Post-hoc multi-class calibration is a common approach for providing confidence estimates of deep neural network predictions. Recent work has shown that widely used scaling methods underestimate their calibration error. We propose a shared class-wise (sCW) calibration strategy, sharing one calibrator among similar classes.
  arXiv  Detail & Related papers  (2020-06-23T15:31:59Z)
• Better Classifier Calibration for Small Data Sets [0.0]
  We show how generating more data for calibration is able to improve calibration algorithm performance. The proposed approach adds computational cost but considering that the main use case is with small data sets this extra computational cost stays insignificant.
  arXiv  Detail & Related papers  (2020-02-24T12:27:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.