Conformal Prediction with Partially Labeled Data
- URL: http://arxiv.org/abs/2306.01191v1
- Date: Thu, 1 Jun 2023 23:10:15 GMT
- Title: Conformal Prediction with Partially Labeled Data
- Authors: Alireza Javanmardi, Yusuf Sale, Paul Hofman, Eyke H\"ullermeier
- Abstract summary: We propose a generalization of the conformal prediction procedure that can be applied to set-valued training and calibration data.
We prove the validity of the proposed method and present experimental studies in which it compares favorably to natural baselines.
- Score: 3.895044919159418
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While the predictions produced by conformal prediction are set-valued, the
data used for training and calibration is supposed to be precise. In the
setting of superset learning or learning from partial labels, a variant of
weakly supervised learning, it is exactly the other way around: training data
is possibly imprecise (set-valued), but the model induced from this data yields
precise predictions. In this paper, we combine the two settings by making
conformal prediction amenable to set-valued training data. We propose a
generalization of the conformal prediction procedure that can be applied to
set-valued training and calibration data. We prove the validity of the proposed
method and present experimental studies in which it compares favorably to
natural baselines.
Related papers
- Split Conformal Prediction under Data Contamination [14.23965125128232]
We study the robustness of split conformal prediction in a data contamination setting.
We quantify the impact of corrupted data on the coverage and efficiency of the constructed sets.
We propose an adjustment in the classification setting which we call Contamination Robust Conformal Prediction.
arXiv Detail & Related papers (2024-07-10T14:33:28Z) - Stochastic Online Conformal Prediction with Semi-Bandit Feedback [29.334511328067777]
We consider the online learning setting, where examples arrive over time, and the goal is to construct prediction sets dynamically.
We propose a novel conformal prediction algorithm targeted at this setting, and prove that it obtains sublinear regret compared to the optimal conformal predictor.
arXiv Detail & Related papers (2024-05-22T00:42:49Z) - Selecting informative conformal prediction sets with false coverage rate control [0.873811641236639]
Conformal methods provide prediction sets for the outcome/label with finite sample coverage for any machine learning predictor.
We consider both the classification and regression settings where the analyst may consider as informative only the sample with prediction sets small enough.
We develop a unified framework for building such informative conformal prediction sets while controlling the false coverage rate (FCR) on the selected sample.
arXiv Detail & Related papers (2024-03-18T22:35:43Z) - Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores.
We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
arXiv Detail & Related papers (2023-02-23T18:57:14Z) - Conformal prediction for the design problem [72.14982816083297]
In many real-world deployments of machine learning, we use a prediction algorithm to choose what data to test next.
In such settings, there is a distinct type of distribution shift between the training and test data.
We introduce a method to quantify predictive uncertainty in such settings.
arXiv Detail & Related papers (2022-02-08T02:59:12Z) - Improved Fine-tuning by Leveraging Pre-training Data: Theory and
Practice [52.11183787786718]
Fine-tuning a pre-trained model on the target data is widely used in many deep learning applications.
Recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy.
We propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task.
arXiv Detail & Related papers (2021-11-24T06:18:32Z) - Private Prediction Sets [72.75711776601973]
Machine learning systems need reliable uncertainty quantification and protection of individuals' privacy.
We present a framework that treats these two desiderata jointly.
We evaluate the method on large-scale computer vision datasets.
arXiv Detail & Related papers (2021-02-11T18:59:11Z) - Robust Validation: Confident Predictions Even When Distributions Shift [19.327409270934474]
We describe procedures for robust predictive inference, where a model provides uncertainty estimates on its predictions rather than point predictions.
We present a method that produces prediction sets (almost exactly) giving the right coverage level for any test distribution in an $f$-divergence ball around the training population.
An essential component of our methodology is to estimate the amount of expected future data shift and build robustness to it.
arXiv Detail & Related papers (2020-08-10T17:09:16Z) - Evaluating Prediction-Time Batch Normalization for Robustness under
Covariate Shift [81.74795324629712]
We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift.
We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness.
The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
arXiv Detail & Related papers (2020-06-19T05:08:43Z) - Balance-Subsampled Stable Prediction [55.13512328954456]
We propose a novel balance-subsampled stable prediction (BSSP) algorithm based on the theory of fractional factorial design.
A design-theoretic analysis shows that the proposed method can reduce the confounding effects among predictors induced by the distribution shift.
Numerical experiments on both synthetic and real-world data sets demonstrate that our BSSP algorithm significantly outperforms the baseline methods for stable prediction across unknown test data.
arXiv Detail & Related papers (2020-06-08T07:01:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.