Priors in Bayesian Deep Learning: A Review

• URL: http://arxiv.org/abs/2105.06868v1
• Date: Fri, 14 May 2021 14:53:30 GMT
• Title: Priors in Bayesian Deep Learning: A Review
• Authors: Vincent Fortuin
• Abstract summary: We highlight the importance of prior choices for Bayesian deep learning. We present an overview of different priors that have been proposed for (deep) Gaussian processes, variational autoencoders, and Bayesian neural networks.
• Score: 4.020523898765405
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While the choice of prior is one of the most critical parts of the Bayesian inference workflow, recent Bayesian deep learning models have often fallen back on uninformative priors, such as standard Gaussians. In this review, we highlight the importance of prior choices for Bayesian deep learning and present an overview of different priors that have been proposed for (deep) Gaussian processes, variational autoencoders, and Bayesian neural networks. We also outline different methods of learning priors for these models from data. We hope to motivate practitioners in Bayesian deep learning to think more carefully about the prior specification for their models and to provide them with some inspiration in this regard.

Related papers

• Amortising Inference and Meta-Learning Priors in Neural Networks [6.79694176040179]
  It is not clear how to represent beliefs about a prediction task by prior distributions over model parameters.<n>We introduce a way to perform per-dataset amortised variational inference.<n>This unique model allows us to study the behaviour of Bayesian neural networks under well-specified priors.
  arXiv  Detail & Related papers  (2026-02-09T15:24:07Z)
• Priors Matter: Addressing Misspecification in Bayesian Deep Q-Learning [12.02900930453346]
  We demonstrate that there is a cold posterior effect in Bayesian deep Q-learning.<n>We show through statistical tests that the common Gaussian likelihood assumption is frequently violated.<n>We argue that developing more suitable likelihoods and priors should be a key focus in future Bayesian reinforcement learning research.
  arXiv  Detail & Related papers  (2025-08-29T10:12:42Z)
• Bayesian Computation in Deep Learning [27.678260738024505]
  We provide an introduction to approximate inference techniques as Bayesian computation methods applied to deep learning models. We present popular computational methods for Bayesian neural networks and deep generative models, explaining their unique challenges in posterior inference as well as the solutions.
  arXiv  Detail & Related papers  (2025-02-25T15:39:33Z)
• Unrolled denoising networks provably learn optimal Bayesian inference [54.79172096306631]
  We prove the first rigorous learning guarantees for neural networks based on unrolling approximate message passing (AMP) For compressed sensing, we prove that when trained on data drawn from a product prior, the layers of the network converge to the same denoisers used in Bayes AMP.
  arXiv  Detail & Related papers  (2024-09-19T17:56:16Z)
• Towards Bayesian Data Selection [0.0]
  Examples include semi-supervised learning, active learning, multi-armed bandits, and Bayesian optimization. We embed this kind of data addition into decision theory by framing data selection as a decision problem. For the illustrative case of self-training in semi-supervised learning, we derive the respective Bayes criterion.
  arXiv  Detail & Related papers  (2024-06-18T12:40:15Z)
• Bayesian Neural Networks with Domain Knowledge Priors [52.80929437592308]
  We propose a framework for integrating general forms of domain knowledge into a BNN prior. We show that BNNs using our proposed domain knowledge priors outperform those with standard priors.
  arXiv  Detail & Related papers  (2024-02-20T22:34:53Z)
• Towards Improved Variational Inference for Deep Bayesian Models [7.841254447222393]
  In this thesis, we explore the use of variational inference (VI) as an approximation. VI is unique in simultaneously approximating the posterior and providing a lower bound to the marginal likelihood. We propose a variational posterior that provides a unified view of inference in Bayesian neural networks and deep Gaussian processes.
  arXiv  Detail & Related papers  (2024-01-23T00:40:20Z)
• Deep networks for system identification: a Survey [56.34005280792013]
  System identification learns mathematical descriptions of dynamic systems from input-output data. Main aim of the identified model is to predict new data from previous observations. We discuss architectures commonly adopted in the literature, like feedforward, convolutional, and recurrent networks.
  arXiv  Detail & Related papers  (2023-01-30T12:38:31Z)
• Bayesian Learning for Neural Networks: an algorithmic survey [95.42181254494287]
  This self-contained survey engages and introduces readers to the principles and algorithms of Bayesian Learning for Neural Networks. It provides an introduction to the topic from an accessible, practical-algorithmic perspective.
  arXiv  Detail & Related papers  (2022-11-21T21:36:58Z)
• Rethinking Bayesian Learning for Data Analysis: The Art of Prior and Inference in Sparsity-Aware Modeling [20.296566563098057]
  Sparse modeling for signal processing and machine learning has been at the focus of scientific research for over two decades. This article reviews some recent advances in incorporating sparsity-promoting priors into three popular data modeling tools.
  arXiv  Detail & Related papers  (2022-05-28T00:43:52Z)
• Pre-Train Your Loss: Easy Bayesian Transfer Learning with Informative Priors [59.93972277761501]
  We show that we can learn highly informative posteriors from the source task, through supervised or self-supervised approaches. This simple modular approach enables significant performance gains and more data-efficient learning on a variety of downstream classification and segmentation tasks.
  arXiv  Detail & Related papers  (2022-05-20T16:19:30Z)
• BNNpriors: A library for Bayesian neural network inference with different prior distributions [32.944046414823916]
  BNNpriors enables state-of-the-art Markov Chain Monte Carlo inference on Bayesian neural networks. It follows a modular approach that eases the design and implementation of new custom priors. It has facilitated foundational discoveries on the nature of the cold posterior effect in Bayesian neural networks.
  arXiv  Detail & Related papers  (2021-05-14T17:11:04Z)
• Exploring Bayesian Deep Learning for Urgent Instructor Intervention Need in MOOC Forums [58.221459787471254]
  Massive Open Online Courses (MOOCs) have become a popular choice for e-learning thanks to their great flexibility. Due to large numbers of learners and their diverse backgrounds, it is taxing to offer real-time support. With the large volume of posts and high workloads for MOOC instructors, it is unlikely that the instructors can identify all learners requiring intervention. This paper explores for the first time Bayesian deep learning on learner-based text posts with two methods: Monte Carlo Dropout and Variational Inference.
  arXiv  Detail & Related papers  (2021-04-26T15:12:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.