Last Layer Empirical Bayes

• URL: http://arxiv.org/abs/2505.15888v1
• Date: Wed, 21 May 2025 18:00:00 GMT
• Title: Last Layer Empirical Bayes
• Authors: Valentin Villecroze, Yixin Wang, Gabriel Loaiza-Ganem,
• Abstract summary: Bayesian neural networks (BNNs) and deep ensembles are among the most prominent approaches to tackle this task.<n>Inspired by recent work showing that the distribution used by ensembles can be understood as a posterior corresponding to a learned data-dependent prior, we propose last layer empirical Bayes (LLEB)<n>LLEB performs on par with existing approaches, highlighting that empirical Bayes is a promising direction for future research in uncertainty quantification.
• Score: 24.94944362766761
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The task of quantifying the inherent uncertainty associated with neural network predictions is a key challenge in artificial intelligence. Bayesian neural networks (BNNs) and deep ensembles are among the most prominent approaches to tackle this task. Both approaches produce predictions by computing an expectation of neural network outputs over some distribution on the corresponding weights; this distribution is given by the posterior in the case of BNNs, and by a mixture of point masses for ensembles. Inspired by recent work showing that the distribution used by ensembles can be understood as a posterior corresponding to a learned data-dependent prior, we propose last layer empirical Bayes (LLEB). LLEB instantiates a learnable prior as a normalizing flow, which is then trained to maximize the evidence lower bound; to retain tractability we use the flow only on the last layer. We show why LLEB is well motivated, and how it interpolates between standard BNNs and ensembles in terms of the strength of the prior that they use. LLEB performs on par with existing approaches, highlighting that empirical Bayes is a promising direction for future research in uncertainty quantification.

Related papers

• PAC-Bayesian risk bounds for fully connected deep neural network with Gaussian priors [0.0]
  We show that fully connected Bayesian neural networks can achieve comparable convergence rates to sparse networks.<n>Our results hold for a broad class of practical activation functions that are Lipschitz continuous.
  arXiv  Detail & Related papers  (2025-05-07T11:42:18Z)
• Unrolled denoising networks provably learn optimal Bayesian inference [54.79172096306631]
  We prove the first rigorous learning guarantees for neural networks based on unrolling approximate message passing (AMP) For compressed sensing, we prove that when trained on data drawn from a product prior, the layers of the network converge to the same denoisers used in Bayes AMP.
  arXiv  Detail & Related papers  (2024-09-19T17:56:16Z)
• Bayesian Neural Networks with Domain Knowledge Priors [52.80929437592308]
  We propose a framework for integrating general forms of domain knowledge into a BNN prior. We show that BNNs using our proposed domain knowledge priors outperform those with standard priors.
  arXiv  Detail & Related papers  (2024-02-20T22:34:53Z)
• Deep Neural Networks Tend To Extrapolate Predictably [51.303814412294514]
  neural network predictions tend to be unpredictable and overconfident when faced with out-of-distribution (OOD) inputs. We observe that neural network predictions often tend towards a constant value as input data becomes increasingly OOD. We show how one can leverage our insights in practice to enable risk-sensitive decision-making in the presence of OOD inputs.
  arXiv  Detail & Related papers  (2023-10-02T03:25:32Z)
• Incorporating Unlabelled Data into Bayesian Neural Networks [48.25555899636015]
  We introduce Self-Supervised Bayesian Neural Networks, which use unlabelled data to learn models with suitable prior predictive distributions. We show that the prior predictive distributions of self-supervised BNNs capture problem semantics better than conventional BNN priors. Our approach offers improved predictive performance over conventional BNNs, especially in low-budget regimes.
  arXiv  Detail & Related papers  (2023-04-04T12:51:35Z)
• Semantic Strengthening of Neuro-Symbolic Learning [85.6195120593625]
  Neuro-symbolic approaches typically resort to fuzzy approximations of a probabilistic objective. We show how to compute this efficiently for tractable circuits. We test our approach on three tasks: predicting a minimum-cost path in Warcraft, predicting a minimum-cost perfect matching, and solving Sudoku puzzles.
  arXiv  Detail & Related papers  (2023-02-28T00:04:22Z)
• Improved uncertainty quantification for neural networks with Bayesian last layer [0.0]
  Uncertainty quantification is an important task in machine learning. We present a reformulation of the log-marginal likelihood of a NN with BLL which allows for efficient training using backpropagation.
  arXiv  Detail & Related papers  (2023-02-21T20:23:56Z)
• Mixtures of Laplace Approximations for Improved Post-Hoc Uncertainty in Deep Learning [24.3370326359959]
  We propose to predict with a Gaussian mixture model posterior that consists of a weighted sum of Laplace approximations of independently trained deep neural networks. We theoretically validate that our approach mitigates overconfidence "far away" from the training data and empirically compare against state-of-the-art baselines on standard uncertainty quantification benchmarks.
  arXiv  Detail & Related papers  (2021-11-05T15:52:48Z)
• BNNpriors: A library for Bayesian neural network inference with different prior distributions [32.944046414823916]
  BNNpriors enables state-of-the-art Markov Chain Monte Carlo inference on Bayesian neural networks. It follows a modular approach that eases the design and implementation of new custom priors. It has facilitated foundational discoveries on the nature of the cold posterior effect in Bayesian neural networks.
  arXiv  Detail & Related papers  (2021-05-14T17:11:04Z)
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration [74.5509794733707]
  We present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.515% on BNNs. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods.
  arXiv  Detail & Related papers  (2021-02-17T18:59:28Z)
• Encoding the latent posterior of Bayesian Neural Networks for uncertainty quantification [10.727102755903616]
  We aim for efficient deep BNNs amenable to complex computer vision architectures. We achieve this by leveraging variational autoencoders (VAEs) to learn the interaction and the latent distribution of the parameters at each network layer. Our approach, Latent-Posterior BNN (LP-BNN), is compatible with the recent BatchEnsemble method, leading to highly efficient (in terms of computation and memory during both training and testing) ensembles.
  arXiv  Detail & Related papers  (2020-12-04T19:50:09Z)
• Exact posterior distributions of wide Bayesian neural networks [51.20413322972014]
  We show that the exact BNN posterior converges (weakly) to the one induced by the GP limit of the prior. For empirical validation, we show how to generate exact samples from a finite BNN on a small dataset via rejection sampling.
  arXiv  Detail & Related papers  (2020-06-18T13:57:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.