Conformal Prediction of Classifiers with Many Classes based on Noisy Labels

• URL: http://arxiv.org/abs/2501.12749v2
• Date: Wed, 13 Aug 2025 16:54:25 GMT
• Title: Conformal Prediction of Classifiers with Many Classes based on Noisy Labels
• Authors: Coby Penso, Jacob Goldberger, Ethan Fetaya,
• Abstract summary: Conformal Prediction (CP) controls the prediction uncertainty of classification systems.<n>We show how we can estimate the noise-free conformal threshold based on the noisy labeled data.<n>We dub our approach Noise-Aware Conformal Prediction (NACP)
• Score: 22.841631892273547
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conformal Prediction (CP) controls the prediction uncertainty of classification systems by producing a small prediction set, ensuring a predetermined probability that the true class lies within this set. This is commonly done by defining a score, based on the model predictions, and setting a threshold on this score using a validation set. In this study, we address the problem of CP calibration when we only have access to a calibration set with noisy labels. We show how we can estimate the noise-free conformal threshold based on the noisy labeled data. We derive a finite sample coverage guarantee for uniform noise that remains effective even in tasks with a large number of classes. We dub our approach Noise-Aware Conformal Prediction (NACP). We illustrate the performance of the proposed results on several standard image classification datasets with a large number of classes.

Related papers

• Effects of label noise on the classification of outlier observations [0.0]
  The study employs both synthetic and real datasets to evaluate the prediction abstention rate for outlier observations.<n>The results indicate that the addition of noise, even in small amounts, can have a significant effect on model performance.
  arXiv  Detail & Related papers  (2025-11-11T22:22:54Z)
• Robust Online Conformal Prediction under Uniform Label Noise [10.059818934854038]
  We investigate the robustness of online conformal prediction under uniform label noise with a known noise rate. We propose Noise Robust Online Conformal Prediction (dubbed NR-OCP) by updating the threshold with a novel robust pinball loss. Our theoretical analysis shows that NR-OCP eliminates the coverage gap in both constant and dynamic learning rate schedules.
  arXiv  Detail & Related papers  (2025-01-30T14:08:26Z)
• Noise-Adaptive Conformal Classification with Marginal Coverage [53.74125453366155]
  We introduce an adaptive conformal inference method capable of efficiently handling deviations from exchangeability caused by random label noise. We validate our method through extensive numerical experiments demonstrating its effectiveness on synthetic and real data sets.
  arXiv  Detail & Related papers  (2025-01-29T23:55:23Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples.<n>We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration [30.039997048267]
  This paper proposes the Rank Calibrated Class-conditional CP (RC3P) algorithm to reduce the prediction set sizes to achieve class-conditional coverage.<n>Experiments on multiple real-world datasets demonstrate that RC3P achieves class-wise coverage and 26.25% reduction in prediction set sizes on average.
  arXiv  Detail & Related papers  (2024-06-10T22:01:34Z)
• A Conformal Prediction Score that is Robust to Label Noise [13.22445242068721]
  We introduce a conformal score that is robust to label noise. The noise-free conformal score is estimated using the noisy labeled data and the noise level. We show that our method outperforms current methods by a large margin, in terms of the average size of the prediction set.
  arXiv  Detail & Related papers  (2024-05-04T12:22:02Z)
• Extracting Clean and Balanced Subset for Noisy Long-tailed Classification [66.47809135771698]
  We develop a novel pseudo labeling method using class prototypes from the perspective of distribution matching. By setting a manually-specific probability measure, we can reduce the side-effects of noisy and long-tailed data simultaneously. Our method can extract this class-balanced subset with clean labels, which brings effective performance gains for long-tailed classification with label noise.
  arXiv  Detail & Related papers  (2024-04-10T07:34:37Z)
• Learning to Correct Noisy Labels for Fine-Grained Entity Typing via Co-Prediction Prompt Tuning [9.885278527023532]
  We introduce Co-Prediction Prompt Tuning for noise correction in FET. We integrate prediction results to recall labeled labels and utilize a differentiated margin to identify inaccurate labels. Experimental results on three widely-used FET datasets demonstrate that our noise correction approach significantly enhances the quality of training samples.
  arXiv  Detail & Related papers  (2023-10-23T06:04:07Z)
• PAC Prediction Sets Under Label Shift [52.30074177997787]
  Prediction sets capture uncertainty by predicting sets of labels rather than individual labels. We propose a novel algorithm for constructing prediction sets with PAC guarantees in the label shift setting. We evaluate our approach on five datasets.
  arXiv  Detail & Related papers  (2023-10-19T17:57:57Z)
• Adaptive conformal classification with noisy labels [22.33857704379073]
  The paper develops novel conformal prediction methods for classification tasks that can automatically adapt to random label contamination in the calibration sample. This is made possible by a precise characterization of the effective coverage inflation suffered by standard conformal inferences in the presence of label contamination. The advantages of the proposed methods are demonstrated through extensive simulations and an application to object classification with the CIFAR-10H image data set.
  arXiv  Detail & Related papers  (2023-09-10T17:35:43Z)
• Class-Conditional Conformal Prediction with Many Classes [60.8189977620604]
  We propose a method called clustered conformal prediction that clusters together classes having "similar" conformal scores. We find that clustered conformal typically outperforms existing methods in terms of class-conditional coverage and set size metrics.
  arXiv  Detail & Related papers  (2023-06-15T17:59:02Z)
• ProTeCt: Prompt Tuning for Taxonomic Open Set Classification [59.59442518849203]
  Few-shot adaptation methods do not fare well in the taxonomic open set (TOS) setting. We propose a prompt tuning technique that calibrates the hierarchical consistency of model predictions. A new Prompt Tuning for Hierarchical Consistency (ProTeCt) technique is then proposed to calibrate classification across label set granularities.
  arXiv  Detail & Related papers  (2023-06-04T02:55:25Z)
• Class Prototype-based Cleaner for Label Noise Learning [73.007001454085]
  Semi-supervised learning methods are current SOTA solutions to the noisy-label learning problem. We propose a simple yet effective solution, named textbfClass textbfPrototype-based label noise textbfCleaner.
  arXiv  Detail & Related papers  (2022-12-21T04:56:41Z)
• Neighborhood Collective Estimation for Noisy Label Identification and Correction [92.20697827784426]
  Learning with noisy labels (LNL) aims at designing strategies to improve model performance and generalization by mitigating the effects of model overfitting to noisy labels. Recent advances employ the predicted label distributions of individual samples to perform noise verification and noisy label correction, easily giving rise to confirmation bias. We propose Neighborhood Collective Estimation, in which the predictive reliability of a candidate sample is re-estimated by contrasting it against its feature-space nearest neighbors.
  arXiv  Detail & Related papers  (2022-08-05T14:47:22Z)
• Uncertainty-Aware Learning Against Label Noise on Imbalanced Datasets [23.4536532321199]
  We propose an Uncertainty-aware Label Correction framework to handle label noise on imbalanced datasets. Inspired by our observations, we propose an Uncertainty-aware Label Correction framework to handle label noise on imbalanced datasets.
  arXiv  Detail & Related papers  (2022-07-12T11:35:55Z)
• S3: Supervised Self-supervised Learning under Label Noise [53.02249460567745]
  In this paper we address the problem of classification in the presence of label noise. In the heart of our method is a sample selection mechanism that relies on the consistency between the annotated label of a sample and the distribution of the labels in its neighborhood in the feature space. Our method significantly surpasses previous methods on both CIFARCIFAR100 with artificial noise and real-world noisy datasets such as WebVision and ANIMAL-10N.
  arXiv  Detail & Related papers  (2021-11-22T15:49:20Z)
• Prototypical Classifier for Robust Class-Imbalanced Learning [64.96088324684683]
  We propose textitPrototypical, which does not require fitting additional parameters given the embedding network. Prototypical produces balanced and comparable predictions for all classes even though the training set is class-imbalanced. We test our method on CIFAR-10LT, CIFAR-100LT and Webvision datasets, observing that Prototypical obtains substaintial improvements compared with state of the arts.
  arXiv  Detail & Related papers  (2021-10-22T01:55:01Z)
• Towards Robustness to Label Noise in Text Classification via Noise Modeling [7.863638253070439]
  Large datasets in NLP suffer from noisy labels, due to erroneous automatic and human annotation procedures. We study the problem of text classification with label noise, and aim to capture this noise through an auxiliary noise model over the classifier.
  arXiv  Detail & Related papers  (2021-01-27T05:41:57Z)
• A Second-Order Approach to Learning with Instance-Dependent Label Noise [58.555527517928596]
  The presence of label noise often misleads the training of deep neural networks. We show that the errors in human-annotated labels are more likely to be dependent on the difficulty levels of tasks.
  arXiv  Detail & Related papers  (2020-12-22T06:36:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.