Function-Space Regularization for Deep Bayesian Classification
- URL: http://arxiv.org/abs/2307.06055v1
- Date: Wed, 12 Jul 2023 10:17:54 GMT
- Title: Function-Space Regularization for Deep Bayesian Classification
- Authors: Jihao Andreas Lin, Joe Watson, Pascal Klink, Jan Peters
- Abstract summary: We apply a Dirichlet prior in predictive space and perform approximate function-space variational inference.
By adapting the inference, the same function-space prior can be combined with different models without affecting model architecture or size.
- Score: 33.63495888167032
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Bayesian deep learning approaches assume model parameters to be latent random
variables and infer posterior distributions to quantify uncertainty, increase
safety and trust, and prevent overconfident and unpredictable behavior.
However, weight-space priors are model-specific, can be difficult to interpret
and are hard to specify. Instead, we apply a Dirichlet prior in predictive
space and perform approximate function-space variational inference. To this
end, we interpret conventional categorical predictions from stochastic neural
network classifiers as samples from an implicit Dirichlet distribution. By
adapting the inference, the same function-space prior can be combined with
different models without affecting model architecture or size. We illustrate
the flexibility and efficacy of such a prior with toy experiments and
demonstrate scalability, improved uncertainty quantification and adversarial
robustness with large-scale image classification experiments.
Related papers
- Exchangeable Sequence Models Can Naturally Quantify Uncertainty Over Latent Concepts [5.095571791233068]
We show that pre-trained sequence models are naturally capable of probabilistic reasoning over exchangeable data points.
A sequence model learns the relationship between observations, which differs from typical Bayesian models.
We show the sequence prediction loss controls the quality of uncertainty quantification.
arXiv Detail & Related papers (2024-08-06T17:16:10Z) - Inflationary Flows: Calibrated Bayesian Inference with Diffusion-Based Models [0.0]
We show how diffusion-based models can be repurposed for performing principled, identifiable Bayesian inference.
We show how such maps can be learned via standard DBM training using a novel noise schedule.
The result is a class of highly expressive generative models, uniquely defined on a low-dimensional latent space.
arXiv Detail & Related papers (2024-07-11T19:58:19Z) - Tractable Function-Space Variational Inference in Bayesian Neural
Networks [72.97620734290139]
A popular approach for estimating the predictive uncertainty of neural networks is to define a prior distribution over the network parameters.
We propose a scalable function-space variational inference method that allows incorporating prior information.
We show that the proposed method leads to state-of-the-art uncertainty estimation and predictive performance on a range of prediction tasks.
arXiv Detail & Related papers (2023-12-28T18:33:26Z) - Calibrating Neural Simulation-Based Inference with Differentiable
Coverage Probability [50.44439018155837]
We propose to include a calibration term directly into the training objective of the neural model.
By introducing a relaxation of the classical formulation of calibration error we enable end-to-end backpropagation.
It is directly applicable to existing computational pipelines allowing reliable black-box posterior inference.
arXiv Detail & Related papers (2023-10-20T10:20:45Z) - Structured Radial Basis Function Network: Modelling Diversity for
Multiple Hypotheses Prediction [51.82628081279621]
Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions.
A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems.
It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
arXiv Detail & Related papers (2023-09-02T01:27:53Z) - Quantification of Predictive Uncertainty via Inference-Time Sampling [57.749601811982096]
We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity.
The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
arXiv Detail & Related papers (2023-08-03T12:43:21Z) - Bayesian Hierarchical Models for Counterfactual Estimation [12.159830463756341]
We propose a probabilistic paradigm to estimate a diverse set of counterfactuals.
We treat the perturbations as random variables endowed with prior distribution functions.
A gradient based sampler with superior convergence characteristics efficiently computes the posterior samples.
arXiv Detail & Related papers (2023-01-21T00:21:11Z) - Dense Uncertainty Estimation [62.23555922631451]
In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation.
We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
arXiv Detail & Related papers (2021-10-13T01:23:48Z) - Multivariate Probabilistic Regression with Natural Gradient Boosting [63.58097881421937]
We propose a Natural Gradient Boosting (NGBoost) approach based on nonparametrically modeling the conditional parameters of the multivariate predictive distribution.
Our method is robust, works out-of-the-box without extensive tuning, is modular with respect to the assumed target distribution, and performs competitively in comparison to existing approaches.
arXiv Detail & Related papers (2021-06-07T17:44:49Z) - Aleatoric uncertainty for Errors-in-Variables models in deep regression [0.48733623015338234]
We show how the concept of Errors-in-Variables can be used in Bayesian deep regression.
We discuss the approach along various simulated and real examples.
arXiv Detail & Related papers (2021-05-19T12:37:02Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.