Conformal Prediction with Partially Labeled Data
- URL: http://arxiv.org/abs/2306.01191v1
- Date: Thu, 1 Jun 2023 23:10:15 GMT
- Title: Conformal Prediction with Partially Labeled Data
- Authors: Alireza Javanmardi, Yusuf Sale, Paul Hofman, Eyke H\"ullermeier
- Abstract summary: We propose a generalization of the conformal prediction procedure that can be applied to set-valued training and calibration data.
We prove the validity of the proposed method and present experimental studies in which it compares favorably to natural baselines.
- Score: 3.895044919159418
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While the predictions produced by conformal prediction are set-valued, the
data used for training and calibration is supposed to be precise. In the
setting of superset learning or learning from partial labels, a variant of
weakly supervised learning, it is exactly the other way around: training data
is possibly imprecise (set-valued), but the model induced from this data yields
precise predictions. In this paper, we combine the two settings by making
conformal prediction amenable to set-valued training data. We propose a
generalization of the conformal prediction procedure that can be applied to
set-valued training and calibration data. We prove the validity of the proposed
method and present experimental studies in which it compares favorably to
natural baselines.
Related papers
- Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
Conformal prediction provides model-agnostic and distribution-free uncertainty quantification.
Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data.
We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
arXiv Detail & Related papers (2024-10-13T15:37:11Z) - Split Conformal Prediction under Data Contamination [14.23965125128232]
We study the robustness of split conformal prediction in a data contamination setting.
We quantify the impact of corrupted data on the coverage and efficiency of the constructed sets.
We propose an adjustment in the classification setting which we call Contamination Robust Conformal Prediction.
arXiv Detail & Related papers (2024-07-10T14:33:28Z) - Self-Calibrating Conformal Prediction [16.606421967131524]
We introduce Self-Calibrating Conformal Prediction to deliver calibrated point predictions alongside prediction intervals with finite-sample validity conditional on these predictions.
We show that our method improves calibrated interval efficiency through model calibration and offers a practical alternative to feature-conditional validity.
arXiv Detail & Related papers (2024-02-11T21:12:21Z) - Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores.
We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
arXiv Detail & Related papers (2023-02-23T18:57:14Z) - Conformal prediction for the design problem [72.14982816083297]
In many real-world deployments of machine learning, we use a prediction algorithm to choose what data to test next.
In such settings, there is a distinct type of distribution shift between the training and test data.
We introduce a method to quantify predictive uncertainty in such settings.
arXiv Detail & Related papers (2022-02-08T02:59:12Z) - Improved Fine-tuning by Leveraging Pre-training Data: Theory and
Practice [52.11183787786718]
Fine-tuning a pre-trained model on the target data is widely used in many deep learning applications.
Recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy.
We propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task.
arXiv Detail & Related papers (2021-11-24T06:18:32Z) - Private Prediction Sets [72.75711776601973]
Machine learning systems need reliable uncertainty quantification and protection of individuals' privacy.
We present a framework that treats these two desiderata jointly.
We evaluate the method on large-scale computer vision datasets.
arXiv Detail & Related papers (2021-02-11T18:59:11Z) - Robust Validation: Confident Predictions Even When Distributions Shift [19.327409270934474]
We describe procedures for robust predictive inference, where a model provides uncertainty estimates on its predictions rather than point predictions.
We present a method that produces prediction sets (almost exactly) giving the right coverage level for any test distribution in an $f$-divergence ball around the training population.
An essential component of our methodology is to estimate the amount of expected future data shift and build robustness to it.
arXiv Detail & Related papers (2020-08-10T17:09:16Z) - Balance-Subsampled Stable Prediction [55.13512328954456]
We propose a novel balance-subsampled stable prediction (BSSP) algorithm based on the theory of fractional factorial design.
A design-theoretic analysis shows that the proposed method can reduce the confounding effects among predictors induced by the distribution shift.
Numerical experiments on both synthetic and real-world data sets demonstrate that our BSSP algorithm significantly outperforms the baseline methods for stable prediction across unknown test data.
arXiv Detail & Related papers (2020-06-08T07:01:38Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.