Few-shot Conformal Prediction with Auxiliary Tasks
- URL: http://arxiv.org/abs/2102.08898v1
- Date: Wed, 17 Feb 2021 17:46:57 GMT
- Title: Few-shot Conformal Prediction with Auxiliary Tasks
- Authors: Adam Fisch, Tal Schuster, Tommi Jaakkola, Regina Barzilay
- Abstract summary: We develop a novel approach to conformal prediction when the target task has limited data available for training.
We obtain substantially tighter prediction sets while maintaining desirable marginal guarantees by casting conformal prediction as a meta-learning paradigm.
We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery.
- Score: 29.034390810078172
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We develop a novel approach to conformal prediction when the target task has
limited data available for training. Conformal prediction identifies a small
set of promising output candidates in place of a single prediction, with
guarantees that the set contains the correct answer with high probability. When
training data is limited, however, the predicted set can easily become unusably
large. In this work, we obtain substantially tighter prediction sets while
maintaining desirable marginal guarantees by casting conformal prediction as a
meta-learning paradigm over exchangeable collections of auxiliary tasks. Our
conformalization algorithm is simple, fast, and agnostic to the choice of
underlying model, learning algorithm, or dataset. We demonstrate the
effectiveness of this approach across a number of few-shot classification and
regression tasks in natural language processing, computer vision, and
computational chemistry for drug discovery.
Related papers
- Uncertainty Quantification for Neurosymbolic Programs via Compositional Conformal Prediction [36.88661670156255]
Conformal prediction has emerged as a promising strategy for quantifying uncertainty in machine learning.
We propose a novel framework for adapting conformal prediction to neurosymbolic programs.
We evaluate our approach on programs that take MNIST and MS-COCO images as input.
arXiv Detail & Related papers (2024-05-24T20:15:53Z) - Stochastic Online Conformal Prediction with Semi-Bandit Feedback [29.334511328067777]
We consider the online learning setting, where examples arrive over time, and the goal is to construct prediction sets dynamically.
We propose a novel conformal prediction algorithm targeted at this setting, and prove that it obtains sublinear regret compared to the optimal conformal predictor.
arXiv Detail & Related papers (2024-05-22T00:42:49Z) - Efficient and Differentiable Conformal Prediction with General Function
Classes [96.74055810115456]
We propose a generalization of conformal prediction to multiple learnable parameters.
We show that it achieves approximate valid population coverage and near-optimal efficiency within class.
Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
arXiv Detail & Related papers (2022-02-22T18:37:23Z) - Non-Clairvoyant Scheduling with Predictions Revisited [77.86290991564829]
In non-clairvoyant scheduling, the task is to find an online strategy for scheduling jobs with a priori unknown processing requirements.
We revisit this well-studied problem in a recently popular learning-augmented setting that integrates (untrusted) predictions in algorithm design.
We show that these predictions have desired properties, admit a natural error measure as well as algorithms with strong performance guarantees.
arXiv Detail & Related papers (2022-02-21T13:18:11Z) - Conformal Prediction Sets with Limited False Positives [43.596058175459746]
We develop a new approach to multi-label conformal prediction in which we aim to output a precise set of promising prediction candidates with a bounded number of incorrect answers.
We demonstrate the effectiveness of this approach across a number of classification tasks in natural language processing, computer vision, and computational chemistry.
arXiv Detail & Related papers (2022-02-15T18:52:33Z) - Private Prediction Sets [72.75711776601973]
Machine learning systems need reliable uncertainty quantification and protection of individuals' privacy.
We present a framework that treats these two desiderata jointly.
We evaluate the method on large-scale computer vision datasets.
arXiv Detail & Related papers (2021-02-11T18:59:11Z) - Distribution-Free, Risk-Controlling Prediction Sets [112.9186453405701]
We show how to generate set-valued predictions from a black-box predictor that control the expected loss on future test points at a user-specified level.
Our approach provides explicit finite-sample guarantees for any dataset by using a holdout set to calibrate the size of the prediction sets.
arXiv Detail & Related papers (2021-01-07T18:59:33Z) - Efficient Conformal Prediction via Cascaded Inference with Expanded
Admission [43.596058175459746]
We present a novel approach for conformal prediction (CP)
We aim to identify a set of promising prediction candidates -- in place of a single prediction.
This set is guaranteed to contain a correct answer with high probability.
arXiv Detail & Related papers (2020-07-06T23:13:07Z) - Video Prediction via Example Guidance [156.08546987158616]
In video prediction tasks, one major challenge is to capture the multi-modal nature of future contents and dynamics.
In this work, we propose a simple yet effective framework that can efficiently predict plausible future states.
arXiv Detail & Related papers (2020-07-03T14:57:24Z) - Predicting Temporal Sets with Deep Neural Networks [50.53727580527024]
We propose an integrated solution based on the deep neural networks for temporal sets prediction.
A unique perspective is to learn element relationship by constructing set-level co-occurrence graph.
We design an attention-based module to adaptively learn the temporal dependency of elements and sets.
arXiv Detail & Related papers (2020-06-20T03:29:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.