Conformal Prediction for Deep Classifier via Label Ranking

• URL: http://arxiv.org/abs/2310.06430v2
• Date: Thu, 6 Jun 2024 13:52:04 GMT
• Title: Conformal Prediction for Deep Classifier via Label Ranking
• Authors: Jianguo Huang, Huajun Xi, Linjun Zhang, Huaxiu Yao, Yue Qiu, Hongxin Wei,
• Abstract summary: Conformal prediction is a statistical framework that generates prediction sets with a desired coverage guarantee. We propose a novel algorithm named $textitSorted Adaptive Prediction Sets$ (SAPS) SAPS discards all the probability values except for the maximum softmax probability.
• Score: 29.784336674173616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conformal prediction is a statistical framework that generates prediction sets containing ground-truth labels with a desired coverage guarantee. The predicted probabilities produced by machine learning models are generally miscalibrated, leading to large prediction sets in conformal prediction. To address this issue, we propose a novel algorithm named $\textit{Sorted Adaptive Prediction Sets}$ (SAPS), which discards all the probability values except for the maximum softmax probability. The key idea behind SAPS is to minimize the dependence of the non-conformity score on the probability values while retaining the uncertainty information. In this manner, SAPS can produce compact prediction sets and communicate instance-wise uncertainty. Extensive experiments validate that SAPS not only lessens the prediction sets but also broadly enhances the conditional coverage rate of prediction sets.

Related papers

• Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering [55.15192437680943]
  Generative models lack rigorous statistical guarantees for their outputs. We propose a sequential conformal prediction method producing prediction sets that satisfy a rigorous statistical guarantee. This guarantee states that with high probability, the prediction sets contain at least one admissible (or valid) example.
  arXiv  Detail & Related papers  (2024-10-02T15:26:52Z)
• Spatial-Aware Conformal Prediction for Trustworthy Hyperspectral Image Classification [39.71307720326761]
  Hyperspectral image (HSI) classification involves assigning unique labels to each pixel to identify various land cover categories. Deep classifiers have achieved high predictive accuracy in this field, but they lack the ability to quantify confidence in their predictions. We introduce Spatial-Aware Conformal Prediction (textttSACP), a conformal prediction framework specifically designed for HSI data.
  arXiv  Detail & Related papers  (2024-09-02T13:11:38Z)
• Towards Human-AI Complementarity with Prediction Sets [14.071862670474832]
  Decision support systems based on prediction sets have proven to be effective at helping human experts solve classification tasks. We show that the prediction sets constructed using conformal prediction are, in general, suboptimal in terms of average accuracy. We introduce a greedy algorithm that, for a large class of expert models and non-optimal scores, is guaranteed to find prediction sets that provably offer equal or greater performance.
  arXiv  Detail & Related papers  (2024-05-27T18:00:00Z)
• PAC Prediction Sets Under Label Shift [52.30074177997787]
  Prediction sets capture uncertainty by predicting sets of labels rather than individual labels. We propose a novel algorithm for constructing prediction sets with PAC guarantees in the label shift setting. We evaluate our approach on five datasets.
  arXiv  Detail & Related papers  (2023-10-19T17:57:57Z)
• Post-selection Inference for Conformal Prediction: Trading off Coverage for Precision [0.0]
  Traditionally, conformal prediction inference requires a data-independent specification of miscoverage level. We develop simultaneous conformal inference to account for data-dependent miscoverage levels.
  arXiv  Detail & Related papers  (2023-04-12T20:56:43Z)
• Probabilistic Conformal Prediction Using Conditional Random Samples [73.26753677005331]
  PCP is a predictive inference algorithm that estimates a target variable by a discontinuous predictive set. It is efficient and compatible with either explicit or implicit conditional generative models.
  arXiv  Detail & Related papers  (2022-06-14T03:58:03Z)
• Taming Overconfident Prediction on Unlabeled Data from Hindsight [50.9088560433925]
  Minimizing prediction uncertainty on unlabeled data is a key factor to achieve good performance in semi-supervised learning. This paper proposes a dual mechanism, named ADaptive Sharpening (ADS), which first applies a soft-threshold to adaptively mask out determinate and negligible predictions. ADS significantly improves the state-of-the-art SSL methods by making it a plug-in.
  arXiv  Detail & Related papers  (2021-12-15T15:17:02Z)
• Private Prediction Sets [72.75711776601973]
  Machine learning systems need reliable uncertainty quantification and protection of individuals' privacy. We present a framework that treats these two desiderata jointly. We evaluate the method on large-scale computer vision datasets.
  arXiv  Detail & Related papers  (2021-02-11T18:59:11Z)
• Efficient Conformal Prediction via Cascaded Inference with Expanded Admission [43.596058175459746]
  We present a novel approach for conformal prediction (CP) We aim to identify a set of promising prediction candidates -- in place of a single prediction. This set is guaranteed to contain a correct answer with high probability.
  arXiv  Detail & Related papers  (2020-07-06T23:13:07Z)
• AutoCP: Automated Pipelines for Accurate Prediction Intervals [84.16181066107984]
  This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP) Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.
  arXiv  Detail & Related papers  (2020-06-24T23:13:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.