On the Out-of-Distribution Coverage of Combining Split Conformal Prediction and Bayesian Deep Learning

• URL: http://arxiv.org/abs/2311.12688v2
• Date: Thu, 7 Mar 2024 17:00:03 GMT
• Title: On the Out-of-Distribution Coverage of Combining Split Conformal Prediction and Bayesian Deep Learning
• Authors: Paul Scemama, Ariel Kapusta
• Abstract summary: We focus on combining Bayesian deep learning with split conformal prediction and how this combination effects out-of-distribution coverage. Our results suggest that combining Bayesian deep learning models with split conformal prediction can, in some cases, cause unintended consequences such as reducing out-of-distribution coverage.
• Score: 1.131316248570352
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Bayesian deep learning and conformal prediction are two methods that have been used to convey uncertainty and increase safety in machine learning systems. We focus on combining Bayesian deep learning with split conformal prediction and how this combination effects out-of-distribution coverage; particularly in the case of multiclass image classification. We suggest that if the model is generally underconfident on the calibration set, then the resultant conformal sets may exhibit worse out-of-distribution coverage compared to simple predictive credible sets. Conversely, if the model is overconfident on the calibration set, the use of conformal prediction may improve out-of-distribution coverage. We evaluate prediction sets as a result of combining split conformal methods and neural networks trained with (i) stochastic gradient descent, (ii) deep ensembles, and (iii) mean-field variational inference. Our results suggest that combining Bayesian deep learning models with split conformal prediction can, in some cases, cause unintended consequences such as reducing out-of-distribution coverage.

Related papers

• Can Bayesian Neural Networks Make Confident Predictions? [0.0]
  We show that under a discretized prior for the inner layer weights, we can exactly characterize the posterior predictive distribution as a Gaussian mixture. We also characterize the capacity of a model to "learn from data" by evaluating contraction of the posterior predictive in different scaling regimes.
  arXiv  Detail & Related papers  (2025-01-20T22:36:28Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples. We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Semiparametric conformal prediction [79.6147286161434]
  Risk-sensitive applications require well-calibrated prediction sets over multiple, potentially correlated target variables. We treat the scores as random vectors and aim to construct the prediction set accounting for their joint correlation structure. We report desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
  Conformal prediction provides model-agnostic and distribution-free uncertainty quantification. Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data. We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
  arXiv  Detail & Related papers  (2024-10-13T15:37:11Z)
• Split Conformal Prediction under Data Contamination [14.23965125128232]
  We study the robustness of split conformal prediction in a data contamination setting. We quantify the impact of corrupted data on the coverage and efficiency of the constructed sets. We propose an adjustment in the classification setting which we call Contamination Robust Conformal Prediction.
  arXiv  Detail & Related papers  (2024-07-10T14:33:28Z)
• Conformalized Late Fusion Multi-View Learning [18.928543069018865]
  Uncertainty quantification for multi-view learning is motivated by the increasing use of multi-view data in scientific problems. A common variant of multi-view learning is late fusion: train separate predictors on individual views and combine them after single-view predictions are available. We propose a novel methodology, Multi-View Conformal Prediction (MVCP), where conformal prediction is instead performed separately on the single-view predictors and only fused subsequently.
  arXiv  Detail & Related papers  (2024-05-25T14:11:01Z)
• Conformalized-DeepONet: A Distribution-Free Framework for Uncertainty Quantification in Deep Operator Networks [7.119066725173193]
  We use conformal prediction to obtain confidence prediction intervals with coverage guarantees for Deep Operator Network (DeepONet) regression. We design a novel Quantile-DeepONet that allows for a more natural use of split conformal prediction. We demonstrate the effectiveness of the proposed methods using various ordinary, partial differential equation numerical examples.
  arXiv  Detail & Related papers  (2024-02-23T16:07:39Z)
• Domain-adaptive and Subgroup-specific Cascaded Temperature Regression for Out-of-distribution Calibration [16.930766717110053]
  We propose a novel meta-set-based cascaded temperature regression method for post-hoc calibration. We partition each meta-set into subgroups based on predicted category and confidence level, capturing diverse uncertainties. A regression network is then trained to derive category-specific and confidence-level-specific scaling, achieving calibration across meta-sets.
  arXiv  Detail & Related papers  (2024-02-14T14:35:57Z)
• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• Distribution-Free Finite-Sample Guarantees and Split Conformal Prediction [0.0]
  split conformal prediction represents a promising avenue to obtain finite-sample guarantees under minimal distribution-free assumptions. We highlight the connection between split conformal prediction and classical tolerance predictors developed in the 1940s.
  arXiv  Detail & Related papers  (2022-10-26T14:12:24Z)
• Predictive Inference with Feature Conformal Prediction [80.77443423828315]
  We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces. From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions. Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
  arXiv  Detail & Related papers  (2022-10-01T02:57:37Z)
• Probabilistic electric load forecasting through Bayesian Mixture Density Networks [70.50488907591463]
  Probabilistic load forecasting (PLF) is a key component in the extended tool-chain required for efficient management of smart energy grids. We propose a novel PLF approach, framed on Bayesian Mixture Density Networks. To achieve reliable and computationally scalable estimators of the posterior distributions, both Mean Field variational inference and deep ensembles are integrated.
  arXiv  Detail & Related papers  (2020-12-23T16:21:34Z)
• Regularizing Class-wise Predictions via Self-knowledge Distillation [80.76254453115766]
  We propose a new regularization method that penalizes the predictive distribution between similar samples. This results in regularizing the dark knowledge (i.e., the knowledge on wrong predictions) of a single network. Our experimental results on various image classification tasks demonstrate that the simple yet powerful method can significantly improve the generalization ability.
  arXiv  Detail & Related papers  (2020-03-31T06:03:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.