Class-Conditional Conformal Prediction with Many Classes
- URL: http://arxiv.org/abs/2306.09335v2
- Date: Fri, 27 Oct 2023 21:24:44 GMT
- Title: Class-Conditional Conformal Prediction with Many Classes
- Authors: Tiffany Ding, Anastasios N. Angelopoulos, Stephen Bates, Michael I.
Jordan, Ryan J. Tibshirani
- Abstract summary: We propose a method called clustered conformal prediction that clusters together classes having "similar" conformal scores.
We find that clustered conformal typically outperforms existing methods in terms of class-conditional coverage and set size metrics.
- Score: 60.8189977620604
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Standard conformal prediction methods provide a marginal coverage guarantee,
which means that for a random test point, the conformal prediction set contains
the true label with a user-specified probability. In many classification
problems, we would like to obtain a stronger guarantee--that for test points of
a specific class, the prediction set contains the true label with the same
user-chosen probability. For the latter goal, existing conformal prediction
methods do not work well when there is a limited amount of labeled data per
class, as is often the case in real applications where the number of classes is
large. We propose a method called clustered conformal prediction that clusters
together classes having "similar" conformal scores and performs conformal
prediction at the cluster level. Based on empirical evaluation across four
image data sets with many (up to 1000) classes, we find that clustered
conformal typically outperforms existing methods in terms of class-conditional
coverage and set size metrics.
Related papers
- Posterior Conformal Prediction [11.576645675675096]
Conformal prediction is a popular technique for constructing prediction intervals with distribution-free coverage guarantees.
This article introduces a new method, posterior conformal prediction (PCP), which generates prediction intervals with both marginal and approximate conditional validity.
We evaluate the performance of PCP on diverse datasets from socio-economic, scientific and healthcare applications.
arXiv Detail & Related papers (2024-09-29T14:09:07Z) - Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration [30.039997048267]
This paper proposes the Rank Calibrated Class-conditional CP (RC3P) algorithm to reduce the prediction set sizes to achieve class-conditional coverage.
Experiments on multiple real-world datasets demonstrate that RC3P achieves class-wise coverage and 26.25% reduction in prediction set sizes on average.
arXiv Detail & Related papers (2024-06-10T22:01:34Z) - Distribution-free Conformal Prediction for Ordinal Classification [0.0]
Ordinal classification is common in real applications where the target variable has natural ordering among the class labels.
New conformal prediction methods are developed for constructing contiguous and non-contiguous prediction sets.
arXiv Detail & Related papers (2024-04-25T13:49:59Z) - Anomaly Detection using Ensemble Classification and Evidence Theory [62.997667081978825]
We present a novel approach for novel detection using ensemble classification and evidence theory.
A pool selection strategy is presented to build a solid ensemble classifier.
We use uncertainty for the anomaly detection approach.
arXiv Detail & Related papers (2022-12-23T00:50:41Z) - Parametric Classification for Generalized Category Discovery: A Baseline
Study [70.73212959385387]
Generalized Category Discovery (GCD) aims to discover novel categories in unlabelled datasets using knowledge learned from labelled samples.
We investigate the failure of parametric classifiers, verify the effectiveness of previous design choices when high-quality supervision is available, and identify unreliable pseudo-labels as a key problem.
We propose a simple yet effective parametric classification method that benefits from entropy regularisation, achieves state-of-the-art performance on multiple GCD benchmarks and shows strong robustness to unknown class numbers.
arXiv Detail & Related papers (2022-11-21T18:47:11Z) - Learning Acceptance Regions for Many Classes with Anomaly Detection [19.269724165953274]
Many existing set-valued classification methods do not consider the possibility that a new class that never appeared in the training data appears in the test data.
We propose a Generalized Prediction Set (GPS) approach to estimate the acceptance regions while considering the possibility of a new class in the test data.
Unlike previous methods, the proposed method achieves a good balance between accuracy, efficiency, and anomaly detection rate.
arXiv Detail & Related papers (2022-09-20T19:40:33Z) - Practical Adversarial Multivalid Conformal Prediction [27.179891682629183]
We give a generic conformal prediction method for sequential prediction.
It achieves target empirical coverage guarantees against adversarially chosen data.
It is computationally lightweight -- comparable to split conformal prediction.
arXiv Detail & Related papers (2022-06-02T14:33:00Z) - When in Doubt: Improving Classification Performance with Alternating
Normalization [57.39356691967766]
We introduce Classification with Alternating Normalization (CAN), a non-parametric post-processing step for classification.
CAN improves classification accuracy for challenging examples by re-adjusting their predicted class probability distribution.
We empirically demonstrate its effectiveness across a diverse set of classification tasks.
arXiv Detail & Related papers (2021-09-28T02:55:42Z) - Cautious Active Clustering [79.23797234241471]
We consider the problem of classification of points sampled from an unknown probability measure on a Euclidean space.
Our approach is to consider the unknown probability measure as a convex combination of the conditional probabilities for each class.
arXiv Detail & Related papers (2020-08-03T23:47:31Z) - Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
Machine learning algorithms are susceptible to data poisoning attacks.
We present a unifying view of randomized smoothing over arbitrary functions.
We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
arXiv Detail & Related papers (2020-02-07T21:28:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.