URSABench: Comprehensive Benchmarking of Approximate Bayesian Inference
Methods for Deep Neural Networks
- URL: http://arxiv.org/abs/2007.04466v1
- Date: Wed, 8 Jul 2020 22:51:28 GMT
- Title: URSABench: Comprehensive Benchmarking of Approximate Bayesian Inference
Methods for Deep Neural Networks
- Authors: Meet P. Vadera, Adam D. Cobb, Brian Jalaian, Benjamin M. Marlin
- Abstract summary: Deep learning methods continue to improve in predictive accuracy on a wide range of application domains.
Recent advances in approximate Bayesian inference hold significant promise for addressing these concerns.
We describe initial work on the development ofURSABench, an open-source suite of bench-marking tools for comprehensive assessment of approximate Bayesian inference methods.
- Score: 15.521736934292354
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: While deep learning methods continue to improve in predictive accuracy on a
wide range of application domains, significant issues remain with other aspects
of their performance including their ability to quantify uncertainty and their
robustness. Recent advances in approximate Bayesian inference hold significant
promise for addressing these concerns, but the computational scalability of
these methods can be problematic when applied to large-scale models. In this
paper, we describe initial work on the development ofURSABench(the Uncertainty,
Robustness, Scalability, and Accu-racy Benchmark), an open-source suite of
bench-marking tools for comprehensive assessment of approximate Bayesian
inference methods with a focus on deep learning-based classification tasks
Related papers
- Deep Learning-Based Object Pose Estimation: A Comprehensive Survey [73.74933379151419]
We discuss the recent advances in deep learning-based object pose estimation.
Our survey also covers multiple input data modalities, degrees-of-freedom of output poses, object properties, and downstream tasks.
arXiv Detail & Related papers (2024-05-13T14:44:22Z) - Tractable Function-Space Variational Inference in Bayesian Neural
Networks [72.97620734290139]
A popular approach for estimating the predictive uncertainty of neural networks is to define a prior distribution over the network parameters.
We propose a scalable function-space variational inference method that allows incorporating prior information.
We show that the proposed method leads to state-of-the-art uncertainty estimation and predictive performance on a range of prediction tasks.
arXiv Detail & Related papers (2023-12-28T18:33:26Z) - Beyond Deep Ensembles: A Large-Scale Evaluation of Bayesian Deep
Learning under Distribution Shift [19.945634052291542]
We evaluate modern BDL algorithms on real-world datasets from the WILDS collection containing challenging classification and regression tasks.
We compare the algorithms on a wide range of large, convolutional and transformer-based neural network architectures.
We provide the first systematic evaluation of BDL for fine-tuning large pre-trained models.
arXiv Detail & Related papers (2023-06-21T14:36:03Z) - Uncertainty Estimation by Fisher Information-based Evidential Deep
Learning [61.94125052118442]
Uncertainty estimation is a key factor that makes deep learning reliable in practical applications.
We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL)
In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
arXiv Detail & Related papers (2023-03-03T16:12:59Z) - BayesCap: Bayesian Identity Cap for Calibrated Uncertainty in Frozen
Neural Networks [50.15201777970128]
We propose BayesCap that learns a Bayesian identity mapping for the frozen model, allowing uncertainty estimation.
BayesCap is a memory-efficient method that can be trained on a small fraction of the original dataset.
We show the efficacy of our method on a wide variety of tasks with a diverse set of architectures.
arXiv Detail & Related papers (2022-07-14T12:50:09Z) - Quantitative performance evaluation of Bayesian neural networks [0.0]
Despite the growing litterature about uncertainty in deep learning, the quality of the uncertainty estimates remains an open question.
In this work, we attempt to assess the performance of several algorithms on sampling and regression tasks.
arXiv Detail & Related papers (2022-06-08T06:56:50Z) - (De-)Randomized Smoothing for Decision Stump Ensembles [5.161531917413708]
Tree-based models are used in many high-stakes application domains such as finance and medicine.
We propose deterministic smoothing for decision stump ensembles.
We obtain deterministic robustness certificates, even jointly over numerical and categorical features.
arXiv Detail & Related papers (2022-05-27T11:23:50Z) - Evaluating Predictive Distributions: Does Bayesian Deep Learning Work? [45.290773422944866]
Posterior predictive distributions quantify uncertainties ignored by point estimates.
This paper introduces textitThe Neural Testbed, which provides tools for the systematic evaluation of agents that generate such predictions.
arXiv Detail & Related papers (2021-10-09T18:54:02Z) - $\beta$-Cores: Robust Large-Scale Bayesian Data Summarization in the
Presence of Outliers [14.918826474979587]
The quality of classic Bayesian inference depends critically on whether observations conform with the assumed data generating model.
We propose a variational inference method that, in a principled way, can simultaneously scale to large datasets.
We illustrate the applicability of our approach in diverse simulated and real datasets, and various statistical models.
arXiv Detail & Related papers (2020-08-31T13:47:12Z) - Provable tradeoffs in adversarially robust classification [96.48180210364893]
We develop and leverage new tools, including recent breakthroughs from probability theory on robust isoperimetry.
Our results reveal fundamental tradeoffs between standard and robust accuracy that grow when data is imbalanced.
arXiv Detail & Related papers (2020-06-09T09:58:19Z) - Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep
Learning [70.72363097550483]
In this study, we focus on in-domain uncertainty for image classification.
To provide more insight in this study, we introduce the deep ensemble equivalent score (DEE)
arXiv Detail & Related papers (2020-02-15T23:28:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.