Distribution-Free, Risk-Controlling Prediction Sets
- URL: http://arxiv.org/abs/2101.02703v2
- Date: Sat, 30 Jan 2021 03:48:34 GMT
- Title: Distribution-Free, Risk-Controlling Prediction Sets
- Authors: Stephen Bates and Anastasios Angelopoulos and Lihua Lei and Jitendra
Malik and Michael I. Jordan
- Abstract summary: We show how to generate set-valued predictions from a black-box predictor that control the expected loss on future test points at a user-specified level.
Our approach provides explicit finite-sample guarantees for any dataset by using a holdout set to calibrate the size of the prediction sets.
- Score: 112.9186453405701
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While improving prediction accuracy has been the focus of machine learning in
recent years, this alone does not suffice for reliable decision-making.
Deploying learning systems in consequential settings also requires calibrating
and communicating the uncertainty of predictions. To convey instance-wise
uncertainty for prediction tasks, we show how to generate set-valued
predictions from a black-box predictor that control the expected loss on future
test points at a user-specified level. Our approach provides explicit
finite-sample guarantees for any dataset by using a holdout set to calibrate
the size of the prediction sets. This framework enables simple,
distribution-free, rigorous error control for many tasks, and we demonstrate it
in five large-scale machine learning problems: (1) classification problems
where some mistakes are more costly than others; (2) multi-label
classification, where each observation has multiple associated labels; (3)
classification problems where the labels have a hierarchical structure; (4)
image segmentation, where we wish to predict a set of pixels containing an
object of interest; and (5) protein structure prediction. Lastly, we discuss
extensions to uncertainty quantification for ranking, metric learning and
distributionally robust learning.
Related papers
- Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
Conformal prediction provides model-agnostic and distribution-free uncertainty quantification.
Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data.
We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
arXiv Detail & Related papers (2024-10-13T15:37:11Z) - Trustworthy Classification through Rank-Based Conformal Prediction Sets [9.559062601251464]
We propose a novel conformal prediction method that employs a rank-based score function suitable for classification models.
Our approach constructs prediction sets that achieve the desired coverage rate while managing their size.
Our contributions include a novel conformal prediction method, theoretical analysis, and empirical evaluation.
arXiv Detail & Related papers (2024-07-05T10:43:41Z) - Uncertainty Quantification for Neurosymbolic Programs via Compositional Conformal Prediction [36.88661670156255]
Conformal prediction has emerged as a promising strategy for quantifying uncertainty in machine learning.
We propose a novel framework for adapting conformal prediction to neurosymbolic programs.
We evaluate our approach on programs that take MNIST and MS-COCO images as input.
arXiv Detail & Related papers (2024-05-24T20:15:53Z) - Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores.
We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
arXiv Detail & Related papers (2023-02-23T18:57:14Z) - Query-Adaptive Predictive Inference with Partial Labels [0.0]
We propose a new methodology to construct predictive sets using only partially labeled data on top of black-box predictive models.
Our experiments highlight the validity of our predictive set construction as well as the attractiveness of a more flexible user-dependent loss framework.
arXiv Detail & Related papers (2022-06-15T01:48:42Z) - Efficient and Differentiable Conformal Prediction with General Function
Classes [96.74055810115456]
We propose a generalization of conformal prediction to multiple learnable parameters.
We show that it achieves approximate valid population coverage and near-optimal efficiency within class.
Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
arXiv Detail & Related papers (2022-02-22T18:37:23Z) - Conformal prediction for the design problem [72.14982816083297]
In many real-world deployments of machine learning, we use a prediction algorithm to choose what data to test next.
In such settings, there is a distinct type of distribution shift between the training and test data.
We introduce a method to quantify predictive uncertainty in such settings.
arXiv Detail & Related papers (2022-02-08T02:59:12Z) - Predictive Inference with Weak Supervision [3.1925030748447747]
We bridge the gap between partial supervision and validation by developing a conformal prediction framework.
We introduce a new notion of coverage and predictive validity, then develop several application scenarios.
We corroborate the hypothesis that the new coverage definition allows for tighter and more informative (but valid) confidence sets.
arXiv Detail & Related papers (2022-01-20T17:26:52Z) - Few-shot Conformal Prediction with Auxiliary Tasks [29.034390810078172]
We develop a novel approach to conformal prediction when the target task has limited data available for training.
We obtain substantially tighter prediction sets while maintaining desirable marginal guarantees by casting conformal prediction as a meta-learning paradigm.
We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery.
arXiv Detail & Related papers (2021-02-17T17:46:57Z) - Private Prediction Sets [72.75711776601973]
Machine learning systems need reliable uncertainty quantification and protection of individuals' privacy.
We present a framework that treats these two desiderata jointly.
We evaluate the method on large-scale computer vision datasets.
arXiv Detail & Related papers (2021-02-11T18:59:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.