Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification

• URL: http://arxiv.org/abs/2304.04042v1
• Date: Sat, 8 Apr 2023 15:25:12 GMT
• Title: Deep Anti-Regularized Ensembles provide reliable out-of-distribution uncertainty quantification
• Authors: Antoine de Mathelin, Francois Deheeger, Mathilde Mougeot, Nicolas Vayatis
• Abstract summary: Deep ensemble often return overconfident estimates outside the training domain. We show that an ensemble of networks with large weights fitting the training data are likely to meet these two objectives. We derive a theoretical framework for this approach and show that the proposed optimization can be seen as a "water-filling" problem.
• Score: 4.750521042508541
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We consider the problem of uncertainty quantification in high dimensional regression and classification for which deep ensemble have proven to be promising methods. Recent observations have shown that deep ensemble often return overconfident estimates outside the training domain, which is a major limitation because shifted distributions are often encountered in real-life scenarios. The principal challenge for this problem is to solve the trade-off between increasing the diversity of the ensemble outputs and making accurate in-distribution predictions. In this work, we show that an ensemble of networks with large weights fitting the training data are likely to meet these two objectives. We derive a simple and practical approach to produce such ensembles, based on an original anti-regularization term penalizing small weights and a control process of the weight increase which maintains the in-distribution loss under an acceptable threshold. The developed approach does not require any out-of-distribution training data neither any trade-off hyper-parameter calibration. We derive a theoretical framework for this approach and show that the proposed optimization can be seen as a "water-filling" problem. Several experiments in both regression and classification settings highlight that Deep Anti-Regularized Ensembles (DARE) significantly improve uncertainty quantification outside the training domain in comparison to recent deep ensembles and out-of-distribution detection methods. All the conducted experiments are reproducible and the source code is available at \url{https://github.com/antoinedemathelin/DARE}.

Related papers

• Error Feedback under $(L_0,L_1)$-Smoothness: Normalization and Momentum [56.37522020675243]
  We provide the first proof of convergence for normalized error feedback algorithms across a wide range of machine learning problems. We show that due to their larger allowable stepsizes, our new normalized error feedback algorithms outperform their non-normalized counterparts on various tasks.
  arXiv  Detail & Related papers  (2024-10-22T10:19:27Z)
• Conformalized-DeepONet: A Distribution-Free Framework for Uncertainty Quantification in Deep Operator Networks [7.119066725173193]
  We use conformal prediction to obtain confidence prediction intervals with coverage guarantees for Deep Operator Network (DeepONet) regression. We design a novel Quantile-DeepONet that allows for a more natural use of split conformal prediction. We demonstrate the effectiveness of the proposed methods using various ordinary, partial differential equation numerical examples.
  arXiv  Detail & Related papers  (2024-02-23T16:07:39Z)
• Hinge-Wasserstein: Estimating Multimodal Aleatoric Uncertainty in Regression Tasks [9.600416563894658]
  We study regression from images to parameter values and here it is common to detect uncertainty by predicting probability distributions. Traditional loss functions lead to poor probability distribution estimates and severe overconfidence, in the absence of full ground truth distributions. We propose hinge-Wasserstein -- a simple improvement of the Wasserstein loss that reduces the penalty for weak secondary modes during training.
  arXiv  Detail & Related papers  (2023-06-01T11:20:09Z)
• Towards Reliable Uncertainty Quantification via Deep Ensembles in Multi-output Regression Task [0.0]
  This study aims to investigate the deep ensemble approach, an approximate Bayesian inference, in the multi-output regression task. A trend towards underestimation of uncertainty as it increases is observed for the first time. We propose the deep ensemble framework that applies the post-hoc calibration method to improve its uncertainty quantification performance.
  arXiv  Detail & Related papers  (2023-03-28T05:10:57Z)
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning [24.3370326359959]
  We propose to predict with a Gaussian mixture model posterior that consists of a weighted sum of Laplace approximations of independently trained deep neural networks. We theoretically validate that our approach mitigates overconfidence "far away" from the training data and empirically compare against state-of-the-art baselines on standard uncertainty quantification benchmarks.
  arXiv  Detail & Related papers  (2021-11-05T15:52:48Z)
• Multivariate Deep Evidential Regression [77.34726150561087]
  A new approach with uncertainty-aware neural networks shows promise over traditional deterministic methods. We discuss three issues with a proposed solution to extract aleatoric and epistemic uncertainties from regression-based neural networks.
  arXiv  Detail & Related papers  (2021-04-13T12:20:18Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Log-Likelihood Ratio Minimizing Flows: Towards Robust and Quantifiable Neural Distribution Alignment [52.02794488304448]
  We propose a new distribution alignment method based on a log-likelihood ratio statistic and normalizing flows. We experimentally verify that minimizing the resulting objective results in domain alignment that preserves the local structure of input domains.
  arXiv  Detail & Related papers  (2020-03-26T22:10:04Z)
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks [65.24701908364383]
  We show that a sufficient condition for a uncertainty on a ReLU network is "to be a bit Bayesian calibrated" We further validate these findings empirically via various standard experiments using common deep ReLU networks and Laplace approximations.
  arXiv  Detail & Related papers  (2020-02-24T08:52:06Z)
• GenDICE: Generalized Offline Estimation of Stationary Values [108.17309783125398]
  We show that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions. The resulting algorithm, GenDICE, is straightforward and effective.
  arXiv  Detail & Related papers  (2020-02-21T00:27:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.