Bayesian neural network priors for edge-preserving inversion
- URL: http://arxiv.org/abs/2112.10663v1
- Date: Mon, 20 Dec 2021 16:39:05 GMT
- Title: Bayesian neural network priors for edge-preserving inversion
- Authors: Chen Li, Matthew Dunlop, Georg Stadler
- Abstract summary: A class of prior distributions based on the output of neural networks with heavy-tailed weights is introduced.
We show theoretically that samples from such priors have desirable discontinuous-like properties even when the network width is finite.
- Score: 3.2046720177804646
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We consider Bayesian inverse problems wherein the unknown state is assumed to
be a function with discontinuous structure a priori. A class of prior
distributions based on the output of neural networks with heavy-tailed weights
is introduced, motivated by existing results concerning the infinite-width
limit of such networks. We show theoretically that samples from such priors
have desirable discontinuous-like properties even when the network width is
finite, making them appropriate for edge-preserving inversion. Numerically we
consider deconvolution problems defined on one- and two-dimensional spatial
domains to illustrate the effectiveness of these priors; MAP estimation,
dimension-robust MCMC sampling and ensemble-based approximations are utilized
to probe the posterior distribution. The accuracy of point estimates is shown
to exceed those obtained from non-heavy tailed priors, and uncertainty
estimates are shown to provide more useful qualitative information.
Related papers
- Unrolled denoising networks provably learn optimal Bayesian inference [54.79172096306631]
We prove the first rigorous learning guarantees for neural networks based on unrolling approximate message passing (AMP)
For compressed sensing, we prove that when trained on data drawn from a product prior, the layers of the network converge to the same denoisers used in Bayes AMP.
arXiv Detail & Related papers (2024-09-19T17:56:16Z) - Tractable Function-Space Variational Inference in Bayesian Neural
Networks [72.97620734290139]
A popular approach for estimating the predictive uncertainty of neural networks is to define a prior distribution over the network parameters.
We propose a scalable function-space variational inference method that allows incorporating prior information.
We show that the proposed method leads to state-of-the-art uncertainty estimation and predictive performance on a range of prediction tasks.
arXiv Detail & Related papers (2023-12-28T18:33:26Z) - Bayesian imaging inverse problem with SA-Roundtrip prior via HMC-pCN
sampler [3.717366858126521]
The selection of the prior distribution is learned from, and therefore an important representation learning of, available prior measurements.
The SA-Roundtrip, a novel deep generative prior, is introduced to enable controlled sampling generation and identify the data's intrinsic dimension.
arXiv Detail & Related papers (2023-10-24T17:16:45Z) - Bayesian Interpolation with Deep Linear Networks [92.1721532941863]
Characterizing how neural network depth, width, and dataset size jointly impact model quality is a central problem in deep learning theory.
We show that linear networks make provably optimal predictions at infinite depth.
We also show that with data-agnostic priors, Bayesian model evidence in wide linear networks is maximized at infinite depth.
arXiv Detail & Related papers (2022-12-29T20:57:46Z) - Robust Estimation for Nonparametric Families via Generative Adversarial
Networks [92.64483100338724]
We provide a framework for designing Generative Adversarial Networks (GANs) to solve high dimensional robust statistics problems.
Our work extend these to robust mean estimation, second moment estimation, and robust linear regression.
In terms of techniques, our proposed GAN losses can be viewed as a smoothed and generalized Kolmogorov-Smirnov distance.
arXiv Detail & Related papers (2022-02-02T20:11:33Z) - Precise characterization of the prior predictive distribution of deep
ReLU networks [45.46732383818331]
We derive a precise characterization of the prior predictive distribution of finite-width ReLU networks with Gaussian weights.
Our results provide valuable guidance on prior design, for instance, controlling the predictive variance with depth- and width-informed priors on the weights of the network.
arXiv Detail & Related papers (2021-06-11T21:21:52Z) - Dimension-robust Function Space MCMC With Neural Network Priors [0.0]
This paper introduces a new prior on functions spaces which scales more favourably in the dimension of the function's domain.
We show that our resulting posterior of the unknown function is amenable to sampling using Hilbert space Markov chain Monte Carlo methods.
We show that our priors are competitive and have distinct advantages over other function space priors.
arXiv Detail & Related papers (2020-12-20T14:52:57Z) - The Ridgelet Prior: A Covariance Function Approach to Prior
Specification for Bayesian Neural Networks [4.307812758854161]
We construct a prior distribution for the parameters of a network that approximates the posited Gaussian process in the output space of the network.
This establishes the property that a Bayesian neural network can approximate any Gaussian process whose covariance function is sufficiently regular.
arXiv Detail & Related papers (2020-10-16T16:39:45Z) - Uncertainty Quantification in Deep Residual Neural Networks [0.0]
Uncertainty quantification is an important and challenging problem in deep learning.
Previous methods rely on dropout layers which are not present in modern deep architectures or batch normalization which is sensitive to batch sizes.
We show that training residual networks using depth can be interpreted as a variational approximation to the posterior weights in neural networks.
arXiv Detail & Related papers (2020-07-09T16:05:37Z) - Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks [65.24701908364383]
We show that a sufficient condition for a uncertainty on a ReLU network is "to be a bit Bayesian calibrated"
We further validate these findings empirically via various standard experiments using common deep ReLU networks and Laplace approximations.
arXiv Detail & Related papers (2020-02-24T08:52:06Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.