From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictions

• URL: http://arxiv.org/abs/2301.11797v1
• Date: Fri, 27 Jan 2023 15:55:01 GMT
• Title: From Classification Accuracy to Proper Scoring Rules: Elicitability of Probabilistic Top List Predictions
• Authors: Johannes Resin
• Abstract summary: I propose a novel type of prediction in classification, which bridges the gap between single-class predictions and predictive distributions. The proposed evaluation metrics are based on symmetric proper scoring rules and admit comparison of various types of predictions.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In the face of uncertainty, the need for probabilistic assessments has long been recognized in the literature on forecasting. In classification, however, comparative evaluation of classifiers often focuses on predictions specifying a single class through the use of simple accuracy measures, which disregard any probabilistic uncertainty quantification. I propose probabilistic top lists as a novel type of prediction in classification, which bridges the gap between single-class predictions and predictive distributions. The probabilistic top list functional is elicitable through the use of strictly consistent evaluation metrics. The proposed evaluation metrics are based on symmetric proper scoring rules and admit comparison of various types of predictions ranging from single-class point predictions to fully specified predictive distributions. The Brier score yields a metric that is particularly well suited for this kind of comparison.

Related papers

• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples. We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Consistency Checks for Language Model Forecasters [54.62507816753479]
  We measure the performance of forecasters in terms of the consistency of their predictions on different logically-related questions. We build an automated evaluation system that generates a set of base questions, instantiates consistency checks from these questions, elicits predictions of the forecaster, and measures the consistency of the predictions.
  arXiv  Detail & Related papers  (2024-12-24T16:51:35Z)
• Conformalized Interval Arithmetic with Symmetric Calibration [9.559062601251464]
  We develop conformal prediction intervals for single target to the prediction interval for sum of multiple targets. We show that our method outperforms existing conformalized approaches as well as non-conformal approaches.
  arXiv  Detail & Related papers  (2024-08-20T15:27:18Z)
• Trustworthy Classification through Rank-Based Conformal Prediction Sets [9.559062601251464]
  We propose a novel conformal prediction method that employs a rank-based score function suitable for classification models. Our approach constructs prediction sets that achieve the desired coverage rate while managing their size. Our contributions include a novel conformal prediction method, theoretical analysis, and empirical evaluation.
  arXiv  Detail & Related papers  (2024-07-05T10:43:41Z)
• Class-Conditional Conformal Prediction with Many Classes [60.8189977620604]
  We propose a method called clustered conformal prediction that clusters together classes having "similar" conformal scores. We find that clustered conformal typically outperforms existing methods in terms of class-conditional coverage and set size metrics.
  arXiv  Detail & Related papers  (2023-06-15T17:59:02Z)
• Predictive Inference with Feature Conformal Prediction [80.77443423828315]
  We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces. From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions. Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
  arXiv  Detail & Related papers  (2022-10-01T02:57:37Z)
• When in Doubt: Improving Classification Performance with Alternating Normalization [57.39356691967766]
  We introduce Classification with Alternating Normalization (CAN), a non-parametric post-processing step for classification. CAN improves classification accuracy for challenging examples by re-adjusting their predicted class probability distribution. We empirically demonstrate its effectiveness across a diverse set of classification tasks.
  arXiv  Detail & Related papers  (2021-09-28T02:55:42Z)
• Performance-Agnostic Fusion of Probabilistic Classifier Outputs [2.4206828137867107]
  We propose a method for combining probabilistic outputs of classifiers to make a single consensus class prediction. Our proposed method works well in situations where accuracy is the performance metric. It does not output calibrated probabilities, so it is not suitable in situations where such probabilities are required for further processing.
  arXiv  Detail & Related papers  (2020-09-01T16:53:29Z)
• Cautious Active Clustering [79.23797234241471]
  We consider the problem of classification of points sampled from an unknown probability measure on a Euclidean space. Our approach is to consider the unknown probability measure as a convex combination of the conditional probabilities for each class.
  arXiv  Detail & Related papers  (2020-08-03T23:47:31Z)
• Classifier uncertainty: evidence, potential impact, and probabilistic treatment [0.0]
  We present an approach to quantify the uncertainty of classification performance metrics based on a probability model of the confusion matrix. We show that uncertainties can be surprisingly large and limit performance evaluation.
  arXiv  Detail & Related papers  (2020-06-19T12:49:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.