A comprehensive review of classifier probability calibration metrics

• URL: http://arxiv.org/abs/2504.18278v1
• Date: Fri, 25 Apr 2025 11:44:44 GMT
• Title: A comprehensive review of classifier probability calibration metrics
• Authors: Richard Oliver Lane,
• Abstract summary: Probabilities or confidence values produced by AI andML models often do not reflect their true accuracy.<n>Probabilities calibration metrics measure the discrepancy between confidence and accuracy.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Probabilities or confidence values produced by artificial intelligence (AI) and machine learning (ML) models often do not reflect their true accuracy, with some models being under or over confident in their predictions. For example, if a model is 80% sure of an outcome, is it correct 80% of the time? Probability calibration metrics measure the discrepancy between confidence and accuracy, providing an independent assessment of model calibration performance that complements traditional accuracy metrics. Understanding calibration is important when the outputs of multiple systems are combined, for assurance in safety or business-critical contexts, and for building user trust in models. This paper provides a comprehensive review of probability calibration metrics for classifier and object detection models, organising them according to a number of different categorisations to highlight their relationships. We identify 82 major metrics, which can be grouped into four classifier families (point-based, bin-based, kernel or curve-based, and cumulative) and an object detection family. For each metric, we provide equations where available, facilitating implementation and comparison by future researchers.

Related papers

• An Entropic Metric for Measuring Calibration of Machine Learning Models [2.467408627377504]
  We show how ECD may be applied to binary classification machine learning models. Our metric distinguishes under- and over-confidence. We demonstrate how this new metric performs on real and simulated data.
  arXiv  Detail & Related papers  (2025-02-20T13:21:18Z)
• Quantifying Uncertainty and Variability in Machine Learning: Confidence Intervals for Quantiles in Performance Metric Distributions [0.17265013728931003]
  Machine learning models are widely used in applications where reliability and robustness are critical.<n>Model evaluation often relies on single-point estimates of performance metrics that fail to capture the inherent variability in model performance.<n>This contribution explores the use of quantiles and confidence intervals to analyze such distributions, providing a more complete understanding of model performance and its uncertainty.
  arXiv  Detail & Related papers  (2025-01-28T13:21:34Z)
• Reassessing How to Compare and Improve the Calibration of Machine Learning Models [7.183341902583164]
  A machine learning model is calibrated if its predicted probability for an outcome matches the observed frequency for that outcome conditional on the model prediction.<n>We show that there exist trivial recalibration approaches that can appear seemingly state-of-the-art unless calibration and prediction metrics are accompanied by additional generalization metrics.
  arXiv  Detail & Related papers  (2024-06-06T13:33:45Z)
• Calibrating Large Language Models with Sample Consistency [76.23956851098598]
  We explore the potential of deriving confidence from the distribution of multiple randomly sampled model generations, via three measures of consistency. Results show that consistency-based calibration methods outperform existing post-hoc approaches. We offer practical guidance on choosing suitable consistency metrics for calibration, tailored to the characteristics of various LMs.
  arXiv  Detail & Related papers  (2024-02-21T16:15:20Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• On the Calibration of Uncertainty Estimation in LiDAR-based Semantic Segmentation [7.100396757261104]
  We propose a metric to measure the confidence calibration quality of a semantic segmentation model with respect to individual classes. We additionally suggest a double use for the method to automatically find label problems to improve the quality of hand- or auto-annotated datasets.
  arXiv  Detail & Related papers  (2023-08-04T10:59:24Z)
• Calibration of Neural Networks [77.34726150561087]
  This paper presents a survey of confidence calibration problems in the context of neural networks. We analyze problem statement, calibration definitions, and different approaches to evaluation. Empirical experiments cover various datasets and models, comparing calibration methods according to different criteria.
  arXiv  Detail & Related papers  (2023-03-19T20:27:51Z)
• Variable Importance Matching for Causal Inference [73.25504313552516]
  We describe a general framework called Model-to-Match that achieves these goals. Model-to-Match uses variable importance measurements to construct a distance metric. We operationalize the Model-to-Match framework with LASSO.
  arXiv  Detail & Related papers  (2023-02-23T00:43:03Z)
• Calibration tests beyond classification [30.616624345970973]
  Most supervised machine learning tasks are subject to irreducible prediction errors. Probabilistic predictive models address this limitation by providing probability distributions that represent a belief over plausible targets. Calibrated models guarantee that the predictions are neither over- nor under-confident.
  arXiv  Detail & Related papers  (2022-10-21T09:49:57Z)
• Calibrate: Interactive Analysis of Probabilistic Model Output [5.444048397001003]
  We present Calibrate, an interactive reliability diagram that is resistant to drawbacks in traditional approaches. We demonstrate the utility of Calibrate through use cases on both real-world and synthetic data.
  arXiv  Detail & Related papers  (2022-07-27T20:01:27Z)
• How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models [95.8037674226622]
  We introduce a 3-dimensional evaluation metric that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity.
  arXiv  Detail & Related papers  (2021-02-17T18:25:30Z)
• Machine learning for causal inference: on the use of cross-fit estimators [77.34726150561087]
  Doubly-robust cross-fit estimators have been proposed to yield better statistical properties. We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE) When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
  arXiv  Detail & Related papers  (2020-04-21T23:09:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.