On the Out-of-Distribution Coverage of Combining Split Conformal
Prediction and Bayesian Deep Learning
- URL: http://arxiv.org/abs/2311.12688v2
- Date: Thu, 7 Mar 2024 17:00:03 GMT
- Title: On the Out-of-Distribution Coverage of Combining Split Conformal
Prediction and Bayesian Deep Learning
- Authors: Paul Scemama, Ariel Kapusta
- Abstract summary: We focus on combining Bayesian deep learning with split conformal prediction and how this combination effects out-of-distribution coverage.
Our results suggest that combining Bayesian deep learning models with split conformal prediction can, in some cases, cause unintended consequences such as reducing out-of-distribution coverage.
- Score: 1.131316248570352
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Bayesian deep learning and conformal prediction are two methods that have
been used to convey uncertainty and increase safety in machine learning
systems. We focus on combining Bayesian deep learning with split conformal
prediction and how this combination effects out-of-distribution coverage;
particularly in the case of multiclass image classification. We suggest that if
the model is generally underconfident on the calibration set, then the
resultant conformal sets may exhibit worse out-of-distribution coverage
compared to simple predictive credible sets. Conversely, if the model is
overconfident on the calibration set, the use of conformal prediction may
improve out-of-distribution coverage. We evaluate prediction sets as a result
of combining split conformal methods and neural networks trained with (i)
stochastic gradient descent, (ii) deep ensembles, and (iii) mean-field
variational inference. Our results suggest that combining Bayesian deep
learning models with split conformal prediction can, in some cases, cause
unintended consequences such as reducing out-of-distribution coverage.
Related papers
- Split Conformal Prediction under Data Contamination [14.23965125128232]
We study the robustness of split conformal prediction in a data contamination setting.
We quantify the impact of corrupted data on the coverage and efficiency of the constructed sets.
We propose an adjustment in the classification setting which we call Contamination Robust Conformal Prediction.
arXiv Detail & Related papers (2024-07-10T14:33:28Z) - Conformalized Late Fusion Multi-View Learning [18.928543069018865]
Uncertainty quantification for multi-view learning is motivated by the increasing use of multi-view data in scientific problems.
A common variant of multi-view learning is late fusion: train separate predictors on individual views and combine them after single-view predictions are available.
We propose a novel methodology, Multi-View Conformal Prediction (MVCP), where conformal prediction is instead performed separately on the single-view predictors and only fused subsequently.
arXiv Detail & Related papers (2024-05-25T14:11:01Z) - Conformalized-DeepONet: A Distribution-Free Framework for Uncertainty
Quantification in Deep Operator Networks [7.119066725173193]
We use conformal prediction to obtain confidence prediction intervals with coverage guarantees for Deep Operator Network (DeepONet) regression.
We design a novel Quantile-DeepONet that allows for a more natural use of split conformal prediction.
We demonstrate the effectiveness of the proposed methods using various ordinary, partial differential equation numerical examples.
arXiv Detail & Related papers (2024-02-23T16:07:39Z) - Domain-adaptive and Subgroup-specific Cascaded Temperature Regression
for Out-of-distribution Calibration [16.930766717110053]
We propose a novel meta-set-based cascaded temperature regression method for post-hoc calibration.
We partition each meta-set into subgroups based on predicted category and confidence level, capturing diverse uncertainties.
A regression network is then trained to derive category-specific and confidence-level-specific scaling, achieving calibration across meta-sets.
arXiv Detail & Related papers (2024-02-14T14:35:57Z) - Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences.
Our method is especially suitable for problems with well-specified likelihoods.
We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
arXiv Detail & Related papers (2023-11-08T00:10:21Z) - Structured Radial Basis Function Network: Modelling Diversity for
Multiple Hypotheses Prediction [51.82628081279621]
Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions.
A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems.
It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
arXiv Detail & Related papers (2023-09-02T01:27:53Z) - Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores.
We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
arXiv Detail & Related papers (2023-02-23T18:57:14Z) - Distribution-Free Finite-Sample Guarantees and Split Conformal
Prediction [0.0]
split conformal prediction represents a promising avenue to obtain finite-sample guarantees under minimal distribution-free assumptions.
We highlight the connection between split conformal prediction and classical tolerance predictors developed in the 1940s.
arXiv Detail & Related papers (2022-10-26T14:12:24Z) - Predictive Inference with Feature Conformal Prediction [80.77443423828315]
We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces.
From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions.
Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
arXiv Detail & Related papers (2022-10-01T02:57:37Z) - Probabilistic electric load forecasting through Bayesian Mixture Density
Networks [70.50488907591463]
Probabilistic load forecasting (PLF) is a key component in the extended tool-chain required for efficient management of smart energy grids.
We propose a novel PLF approach, framed on Bayesian Mixture Density Networks.
To achieve reliable and computationally scalable estimators of the posterior distributions, both Mean Field variational inference and deep ensembles are integrated.
arXiv Detail & Related papers (2020-12-23T16:21:34Z) - Regularizing Class-wise Predictions via Self-knowledge Distillation [80.76254453115766]
We propose a new regularization method that penalizes the predictive distribution between similar samples.
This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network.
Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve the generalization ability.
arXiv Detail & Related papers (2020-03-31T06:03:51Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.