Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration

• URL: http://arxiv.org/abs/2408.02841v1
• Date: Mon, 5 Aug 2024 21:35:51 GMT
• Title: Evaluating Posterior Probabilities: Decision Theory, Proper Scoring Rules, and Calibration
• Authors: Luciana Ferrer, Daniel Ramos,
• Abstract summary: We argue that calibration metrics should play no role in the assessment of posterior quality. We discuss a simple and practical calibration metric, called calibration loss, derived from a decomposition of expected PSRs.
• Score: 10.604555099281173
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Most machine learning classifiers are designed to output posterior probabilities for the classes given the input sample. These probabilities may be used to make the categorical decision on the class of the sample; provided as input to a downstream system; or provided to a human for interpretation. Evaluating the quality of the posteriors generated by these system is an essential problem which was addressed decades ago with the invention of proper scoring rules (PSRs). Unfortunately, much of the recent machine learning literature uses calibration metrics -- most commonly, the expected calibration error (ECE) -- as a proxy to assess posterior performance. The problem with this approach is that calibration metrics reflect only one aspect of the quality of the posteriors, ignoring the discrimination performance. For this reason, we argue that calibration metrics should play no role in the assessment of posterior quality. Expected PSRs should instead be used for this job, preferably normalized for ease of interpretation. In this work, we first give a brief review of PSRs from a practical perspective, motivating their definition using Bayes decision theory. We discuss why expected PSRs provide a principled measure of the quality of a system's posteriors and why calibration metrics are not the right tool for this job. We argue that calibration metrics, while not useful for performance assessment, may be used as diagnostic tools during system development. With this purpose in mind, we discuss a simple and practical calibration metric, called calibration loss, derived from a decomposition of expected PSRs. We compare this metric with the ECE and with the expected score divergence calibration metric from the PSR literature and argue, using theoretical and empirical evidence, that calibration loss is superior to these two metrics.

Related papers

• Optimizing Estimators of Squared Calibration Errors in Classification [2.3020018305241337]
  We propose a mean-squared error-based risk that enables the comparison and optimization of estimators of squared calibration errors. Our approach advocates for a training-validation-testing pipeline when estimating a calibration error.
  arXiv  Detail & Related papers  (2024-10-09T15:58:06Z)
• Orthogonal Causal Calibration [55.28164682911196]
  We prove generic upper bounds on the calibration error of any causal parameter estimate $theta$ with respect to any loss $ell$. We use our bound to analyze the convergence of two sample splitting algorithms for causal calibration.
  arXiv  Detail & Related papers  (2024-06-04T03:35:25Z)
• From Uncertainty to Precision: Enhancing Binary Classifier Performance through Calibration [0.3495246564946556]
  Given that model-predicted scores are commonly seen as event probabilities, calibration is crucial for accurate interpretation. We analyze the sensitivity of various calibration measures to score distortions and introduce a refined metric, the Local Score. We apply these findings in a real-world scenario using Random Forest classifier and regressor to predict credit default while simultaneously measuring calibration.
  arXiv  Detail & Related papers  (2024-02-12T16:55:19Z)
• Calibration by Distribution Matching: Trainable Kernel Calibration Metrics [56.629245030893685]
  We introduce kernel-based calibration metrics that unify and generalize popular forms of calibration for both classification and regression. These metrics admit differentiable sample estimates, making it easy to incorporate a calibration objective into empirical risk minimization. We provide intuitive mechanisms to tailor calibration metrics to a decision task, and enforce accurate loss estimation and no regret decisions.
  arXiv  Detail & Related papers  (2023-10-31T06:19:40Z)
• Beyond calibration: estimating the grouping loss of modern neural networks [68.8204255655161]
  Proper scoring rule theory shows that given the calibration loss, the missing piece to characterize individual errors is the grouping loss. We show that modern neural network architectures in vision and NLP exhibit grouping loss, notably in distribution shifts settings.
  arXiv  Detail & Related papers  (2022-10-28T07:04:20Z)
• Analysis and Comparison of Classification Metrics [12.092755413404245]
  Metrics for measuring the quality of system scores include the area under the ROC curve, equal error rate, cross-entropy, Brier score, and Bayes EC or Bayes risk. We show how to use these metrics to compute a system's calibration loss and compare this metric with the widely-used expected calibration error (ECE)
  arXiv  Detail & Related papers  (2022-09-12T16:06:10Z)
• What is Your Metric Telling You? Evaluating Classifier Calibration under Context-Specific Definitions of Reliability [6.510061176722249]
  We argue that more expressive metrics must be developed that accurately measure calibration error. We use a generalization of Expected Error (ECE) that measure calibration error under different definitions of reliability. We find that: 1) definitions ECE that focus solely on the predicted class fail to accurately measure calibration error under a selection of practically useful definitions of reliability and 2) many common calibration techniques fail to improve calibration performance uniformly across ECE metrics.
  arXiv  Detail & Related papers  (2022-05-23T16:45:02Z)
• Investigation of Different Calibration Methods for Deep Speaker Embedding based Verification Systems [66.61691401921296]
  This paper presents an investigation over several methods of score calibration for deep speaker embedding extractors. An additional focus of this research is to estimate the impact of score normalization on the calibration performance of the system.
  arXiv  Detail & Related papers  (2022-03-28T21:22:22Z)
• Estimating Expected Calibration Errors [1.52292571922932]
  Uncertainty in probabilistics predictions is a key concern when models are used to support human decision making. Most models are not intrinsically well calibrated, meaning that their decision scores are not consistent with posterior probabilities. We build an empirical procedure to quantify the quality of $ECE$ estimators, and use it to decide which estimator should be used in practice for different settings.
  arXiv  Detail & Related papers  (2021-09-08T08:00:23Z)
• Localized Calibration: Metrics and Recalibration [133.07044916594361]
  We propose a fine-grained calibration metric that spans the gap between fully global and fully individualized calibration. We then introduce a localized recalibration method, LoRe, that improves the LCE better than existing recalibration methods.
  arXiv  Detail & Related papers  (2021-02-22T07:22:12Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.