Efficient and Differentiable Conformal Prediction with General Function Classes

• URL: http://arxiv.org/abs/2202.11091v1
• Date: Tue, 22 Feb 2022 18:37:23 GMT
• Title: Efficient and Differentiable Conformal Prediction with General Function Classes
• Authors: Yu Bai, Song Mei, Huan Wang, Yingbo Zhou, Caiming Xiong
• Abstract summary: We propose a generalization of conformal prediction to multiple learnable parameters. We show that it achieves approximate valid population coverage and near-optimal efficiency within class. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
• Score: 96.74055810115456
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Quantifying the data uncertainty in learning tasks is often done by learning a prediction interval or prediction set of the label given the input. Two commonly desired properties for learned prediction sets are \emph{valid coverage} and \emph{good efficiency} (such as low length or low cardinality). Conformal prediction is a powerful technique for learning prediction sets with valid coverage, yet by default its conformalization step only learns a single parameter, and does not optimize the efficiency over more expressive function classes. In this paper, we propose a generalization of conformal prediction to multiple learnable parameters, by considering the constrained empirical risk minimization (ERM) problem of finding the most efficient prediction set subject to valid empirical coverage. This meta-algorithm generalizes existing conformal prediction algorithms, and we show that it achieves approximate valid population coverage and near-optimal efficiency within class, whenever the function class in the conformalization step is low-capacity in a certain sense. Next, this ERM problem is challenging to optimize as it involves a non-differentiable coverage constraint. We develop a gradient-based algorithm for it by approximating the original constrained ERM using differentiable surrogate losses and Lagrangians. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly over existing approaches in several applications such as prediction intervals with improved length, minimum-volume prediction sets for multi-output regression, and label prediction sets for image classification.

Related papers

• Weighted Aggregation of Conformity Scores for Classification [9.559062601251464]
  Conformal prediction is a powerful framework for constructing prediction sets with valid coverage guarantees. We propose a novel approach that combines multiple score functions to improve the performance of conformal predictors.
  arXiv  Detail & Related papers  (2024-07-14T14:58:03Z)
• PAC-Bayes Generalization Certificates for Learned Inductive Conformal Prediction [27.434939269672288]
  We use PAC-Bayes theory to obtain generalization bounds on the coverage and the efficiency of set-valued predictors. We leverage these theoretical results to provide a practical algorithm for using calibration data to fine-tune the parameters of a model and score function. We evaluate the approach on regression and classification tasks, and outperform baselines calibrated using a Hoeffding bound-based PAC guarantee on ICP.
  arXiv  Detail & Related papers  (2023-12-07T19:40:44Z)
• Scalable Bayesian Meta-Learning through Generalized Implicit Gradients [64.21628447579772]
  Implicit Bayesian meta-learning (iBaML) method broadens the scope of learnable priors, but also quantifies the associated uncertainty. Analytical error bounds are established to demonstrate the precision and efficiency of the generalized implicit gradient over the explicit one.
  arXiv  Detail & Related papers  (2023-03-31T02:10:30Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Non-Clairvoyant Scheduling with Predictions Revisited [77.86290991564829]
  In non-clairvoyant scheduling, the task is to find an online strategy for scheduling jobs with a priori unknown processing requirements. We revisit this well-studied problem in a recently popular learning-augmented setting that integrates (untrusted) predictions in algorithm design. We show that these predictions have desired properties, admit a natural error measure as well as algorithms with strong performance guarantees.
  arXiv  Detail & Related papers  (2022-02-21T13:18:11Z)
• Domain-Adjusted Regression or: ERM May Already Learn Features Sufficient for Out-of-Distribution Generalization [52.7137956951533]
  We argue that devising simpler methods for learning predictors on existing features is a promising direction for future research. We introduce Domain-Adjusted Regression (DARE), a convex objective for learning a linear predictor that is provably robust under a new model of distribution shift. Under a natural model, we prove that the DARE solution is the minimax-optimal predictor for a constrained set of test distributions.
  arXiv  Detail & Related papers  (2022-02-14T16:42:16Z)
• Robustification of Online Graph Exploration Methods [59.50307752165016]
  We study a learning-augmented variant of the classical, notoriously hard online graph exploration problem. We propose an algorithm that naturally integrates predictions into the well-known Nearest Neighbor (NN) algorithm.
  arXiv  Detail & Related papers  (2021-12-10T10:02:31Z)
• Optimized conformal classification using gradient descent approximation [0.2538209532048866]
  Conformal predictors allow predictions to be made with a user-defined confidence level. We consider an approach to train the conformal predictor directly with maximum predictive efficiency. We test the method on several real world data sets and find that the method is promising.
  arXiv  Detail & Related papers  (2021-05-24T13:14:41Z)
• Few-shot Conformal Prediction with Auxiliary Tasks [29.034390810078172]
  We develop a novel approach to conformal prediction when the target task has limited data available for training. We obtain substantially tighter prediction sets while maintaining desirable marginal guarantees by casting conformal prediction as a meta-learning paradigm. We demonstrate the effectiveness of this approach across a number of few-shot classification and regression tasks in natural language processing, computer vision, and computational chemistry for drug discovery.
  arXiv  Detail & Related papers  (2021-02-17T17:46:57Z)
• Integrated Optimization of Predictive and Prescriptive Tasks [0.0]
  We propose a new framework directly integrating predictive tasks under prescriptive tasks. We train the parameters of predictive algorithm within a prescription problem via bilevel optimization techniques.
  arXiv  Detail & Related papers  (2021-01-02T02:43:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.