A novel Deep Learning approach for one-step Conformal Prediction approximation

• URL: http://arxiv.org/abs/2207.12377v4
• Date: Mon, 7 Aug 2023 12:02:40 GMT
• Title: A novel Deep Learning approach for one-step Conformal Prediction approximation
• Authors: Julia A. Meister, Khuong An Nguyen, Stelios Kapetanakis, Zhiyuan Luo
• Abstract summary: Conformal Prediction (CP) is a versatile solution that guarantees a maximum error rate given minimal constraints. We propose a novel conformal loss function that approximates the traditionally two-step CP approach in a single step.
• Score: 0.7646713951724009
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Learning predictions with measurable confidence are increasingly desirable for real-world problems, especially in high-risk settings. The Conformal Prediction (CP) framework is a versatile solution that guarantees a maximum error rate given minimal constraints. In this paper, we propose a novel conformal loss function that approximates the traditionally two-step CP approach in a single step. By evaluating and penalising deviations from the stringent expected CP output distribution, a Deep Learning model may learn the direct relationship between the input data and the conformal p-values. We carry out a comprehensive empirical evaluation to show our novel loss function's competitiveness for seven binary and multi-class prediction tasks on five benchmark datasets. On the same datasets, our approach achieves significant training time reductions up to 86% compared to Aggregated Conformal Prediction (ACP), while maintaining comparable approximate validity and predictive efficiency.

Related papers

• Adapting Conformal Prediction to Distribution Shifts Without Labels [16.478151550456804]
  Conformal prediction (CP) enables machine learning models to output prediction sets with guaranteed coverage rate. Our goal is to improve the quality of CP-generated prediction sets using only unlabeled data from the test domain. This is achieved by two new methods called ECP and EACP, that adjust the score function in CP according to the base model's uncertainty on the unlabeled test data.
  arXiv  Detail & Related papers  (2024-06-03T15:16:02Z)
• Loss Shaping Constraints for Long-Term Time Series Forecasting [79.3533114027664]
  We present a Constrained Learning approach for long-term time series forecasting that respects a user-defined upper bound on the loss at each time-step. We propose a practical Primal-Dual algorithm to tackle it, and aims to demonstrate that it exhibits competitive average performance in time series benchmarks, while shaping the errors across the predicted window.
  arXiv  Detail & Related papers  (2024-02-14T18:20:44Z)
• The Efficacy of Pessimism in Asynchronous Q-Learning [17.193902915070506]
  We develop an algorithmic framework that incorporates the principle of pessimism into asynchronous Q-learning. This framework leads to, among other things, improved sample efficiency and enhanced adaptivity in the presence of near-expert data. Our results deliver the first theoretical support for the use of pessimism principle in the presence of Markovian non-i.i.d. data.
  arXiv  Detail & Related papers  (2022-03-14T17:59:01Z)
• Efficient and Differentiable Conformal Prediction with General Function Classes [96.74055810115456]
  We propose a generalization of conformal prediction to multiple learnable parameters. We show that it achieves approximate valid population coverage and near-optimal efficiency within class. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
  arXiv  Detail & Related papers  (2022-02-22T18:37:23Z)
• Adaptive Affinity Loss and Erroneous Pseudo-Label Refinement for Weakly Supervised Semantic Segmentation [48.294903659573585]
  In this paper, we propose to embed affinity learning of multi-stage approaches in a single-stage model. A deep neural network is used to deliver comprehensive semantic information in the training phase. Experiments are conducted on the PASCAL VOC 2012 dataset to evaluate the effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2021-08-03T07:48:33Z)
• Learning Prediction Intervals for Regression: Generalization and Calibration [12.576284277353606]
  We study the generation of prediction intervals in regression for uncertainty quantification. We use a general learning theory to characterize the optimality-feasibility tradeoff that encompasses Lipschitz continuity and VC-subgraph classes. We empirically demonstrate the strengths of our interval generation and calibration algorithms in terms of testing performances compared to existing benchmarks.
  arXiv  Detail & Related papers  (2021-02-26T17:55:30Z)
• Doubly Robust Off-Policy Actor-Critic: Convergence and Optimality [131.45028999325797]
  We develop a doubly robust off-policy AC (DR-Off-PAC) for discounted MDP. DR-Off-PAC adopts a single timescale structure, in which both actor and critics are updated simultaneously with constant stepsize. We study the finite-time convergence rate and characterize the sample complexity for DR-Off-PAC to attain an $epsilon$-accurate optimal policy.
  arXiv  Detail & Related papers  (2021-02-23T18:56:13Z)
• CoinDICE: Off-Policy Confidence Interval Estimation [107.86876722777535]
  We study high-confidence behavior-agnostic off-policy evaluation in reinforcement learning. We show in a variety of benchmarks that the confidence interval estimates are tighter and more accurate than existing methods.
  arXiv  Detail & Related papers  (2020-10-22T12:39:11Z)
• Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited [68.8204255655161]
  We introduce the CORP approach, which generates provably statistically Consistent, Optimally binned, and Reproducible reliability diagrams in an automated way. Corpor is based on non-parametric isotonic regression and implemented via the Pool-adjacent-violators (PAV) algorithm.
  arXiv  Detail & Related papers  (2020-08-07T08:22:26Z)
• PIVEN: A Deep Neural Network for Prediction Intervals with Specific Value Prediction [14.635820704895034]
  We present PIVEN, a deep neural network for producing both a PI and a value prediction. Our approach makes no assumptions regarding data distribution within the PI, making its value prediction more effective for various real-world problems.
  arXiv  Detail & Related papers  (2020-06-09T09:29:58Z)
• Estimating Uncertainty Intervals from Collaborating Networks [15.467208581231848]
  We propose a novel method to capture predictive distributions in regression by defining two neural networks with two distinct loss functions. Specifically, one network approximates the cumulative distribution function, and the second network approximates its inverse. We benchmark CN against several common approaches on two synthetic and six real-world datasets, including forecasting A1c values in diabetic patients from electronic health records.
  arXiv  Detail & Related papers  (2020-02-12T20:10:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.