Variational Bayesian Bow tie Neural Networks with Shrinkage

• URL: http://arxiv.org/abs/2411.11132v3
• Date: Tue, 17 Jun 2025 14:49:16 GMT
• Title: Variational Bayesian Bow tie Neural Networks with Shrinkage
• Authors: Alisa Sheinkman, Sara Wade,
• Abstract summary: We develop a fast, approximate variational inference algorithm that avoids distributional assumptions and independence across layers.<n>We use Polya-Gamma data augmentation tricks, which render a conditionally linear and Gaussian model.
• Score: 0.276240219662896
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the dominant role of deep models in machine learning, limitations persist, including overconfident predictions, susceptibility to adversarial attacks, and underestimation of variability in predictions. The Bayesian paradigm provides a natural framework to overcome such issues and has become the gold standard for uncertainty estimation with deep models, also providing improved accuracy and a framework for tuning critical hyperparameters. However, exact Bayesian inference is challenging, typically involving variational algorithms that impose strong independence and distributional assumptions. Moreover, existing methods are sensitive to the architectural choice of the network. We address these issues by focusing on a stochastic relaxation of the standard feed-forward rectified neural network and using sparsity-promoting priors on the weights of the neural network for increased robustness to architectural design. Thanks to Polya-Gamma data augmentation tricks, which render a conditionally linear and Gaussian model, we derive a fast, approximate variational inference algorithm that avoids distributional assumptions and independence across layers. Suitable strategies to further improve scalability and account for multimodality are considered.

Related papers

• Enhancing Uncertainty Estimation and Interpretability via Bayesian Non-negative Decision Layer [55.66973223528494]
  We develop a Bayesian Non-negative Decision Layer (BNDL), which reformulates deep neural networks as a conditional Bayesian non-negative factor analysis.<n>BNDL can model complex dependencies and provide robust uncertainty estimation.<n>We also offer theoretical guarantees that BNDL can achieve effective disentangled learning.
  arXiv  Detail & Related papers  (2025-05-28T10:23:34Z)
• Certified Neural Approximations of Nonlinear Dynamics [52.79163248326912]
  In safety-critical contexts, the use of neural approximations requires formal bounds on their closeness to the underlying system.<n>We propose a novel, adaptive, and parallelizable verification method based on certified first-order models.
  arXiv  Detail & Related papers  (2025-05-21T13:22:20Z)
• The Architecture and Evaluation of Bayesian Neural Networks [0.276240219662896]
  Despite promising theoretical results, the properties of even the most commonly used posterior approximations are often questioned. The dimensions of modern deep models coupled with the lack of identifiability make Markov chain Monte Carlo expensive and unable to fully explore the posterior. This paper aims to shed some light on this, by considering the computational costs, accuracy and uncertainty quantification in different scenarios including large width and out-of-sample data.
  arXiv  Detail & Related papers  (2025-03-14T18:55:48Z)
• Imitation Learning of MPC with Neural Networks: Error Guarantees and Sparsification [5.260346080244568]
  We present a framework for bounding the approximation error in imitation model predictive controllers utilizing neural networks. We discuss how this method can be used to design a stable neural network controller with performance guarantees.
  arXiv  Detail & Related papers  (2025-01-07T10:18:37Z)
• Deep Learning for Resilient Adversarial Decision Fusion in Byzantine Networks [0.43512163406551996]
  This paper introduces a deep learning-based framework for resilient decision fusion in adversarial multi-sensor networks.<n>The proposed approach employs a deep neural network trained on a globally constructed dataset to generalize across all cases without requiring adaptation.
  arXiv  Detail & Related papers  (2024-12-17T10:02:04Z)
• Alpha-VI DeepONet: A prior-robust variational Bayesian approach for enhancing DeepONets with uncertainty quantification [0.0]
  We introduce a novel deep operator network (DeepONet) framework that incorporates generalised variational inference (GVI) By incorporating Bayesian neural networks as the building blocks for the branch and trunk networks, our framework endows DeepONet with uncertainty quantification. We demonstrate that modifying the variational objective function yields superior results in terms of minimising the mean squared error.
  arXiv  Detail & Related papers  (2024-08-01T16:22:03Z)
• Variational Inference for Bayesian Neural Networks under Model and Parameter Uncertainty [12.211659310564425]
  We apply the concept of model uncertainty as a framework for structural learning in BNNs. We suggest an adaptation of a scalable variational inference approach with reparametrization of marginal inclusion probabilities.
  arXiv  Detail & Related papers  (2023-05-01T16:38:17Z)
• Deep Grey-Box Modeling With Adaptive Data-Driven Models Toward Trustworthy Estimation of Theory-Driven Models [88.63781315038824]
  We present a framework that enables us to analyze a regularizer's behavior empirically with a slight change in the neural net's architecture and the training objective.
  arXiv  Detail & Related papers  (2022-10-24T10:42:26Z)
• Regression modelling of spatiotemporal extreme U.S. wildfires via partially-interpretable neural networks [0.0]
  We propose a new methodological framework for performing extreme quantile regression using artificial neutral networks. We unify linear, and additive, regression methodology with deep learning to create partially-interpretable neural networks.
  arXiv  Detail & Related papers  (2022-08-16T07:42:53Z)
• Variational Bayes Deep Operator Network: A data-driven Bayesian solver for parametric differential equations [0.0]
  We propose Variational Bayes DeepONet (VB-DeepONet) for operator learning. VB-DeepONet uses variational inference to take into account high dimensional posterior distributions.
  arXiv  Detail & Related papers  (2022-06-12T04:20:11Z)
• A Variational Inference Approach to Inverse Problems with Gamma Hyperpriors [60.489902135153415]
  This paper introduces a variational iterative alternating scheme for hierarchical inverse problems with gamma hyperpriors. The proposed variational inference approach yields accurate reconstruction, provides meaningful uncertainty quantification, and is easy to implement.
  arXiv  Detail & Related papers  (2021-11-26T06:33:29Z)
• Bayesian Attention Belief Networks [59.183311769616466]
  Attention-based neural networks have achieved state-of-the-art results on a wide range of tasks. This paper introduces Bayesian attention belief networks, which construct a decoder network by modeling unnormalized attention weights. We show that our method outperforms deterministic attention and state-of-the-art attention in accuracy, uncertainty estimation, generalization across domains, and adversarial attacks.
  arXiv  Detail & Related papers  (2021-06-09T17:46:22Z)
• Multivariate Deep Evidential Regression [77.34726150561087]
  A new approach with uncertainty-aware neural networks shows promise over traditional deterministic methods. We discuss three issues with a proposed solution to extract aleatoric and epistemic uncertainties from regression-based neural networks.
  arXiv  Detail & Related papers  (2021-04-13T12:20:18Z)
• Towards Trustworthy Predictions from Deep Neural Networks with Fast Adversarial Calibration [2.8935588665357077]
  We propose an efficient yet general modelling approach for obtaining well-calibrated, trustworthy probabilities for samples obtained after a domain shift. We introduce a new training strategy combining an entropy-encouraging loss term with an adversarial calibration loss term and demonstrate that this results in well-calibrated and technically trustworthy predictions.
  arXiv  Detail & Related papers  (2020-12-20T13:39:29Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Bayesian Deep Learning and a Probabilistic Perspective of Generalization [56.69671152009899]
  We show that deep ensembles provide an effective mechanism for approximate Bayesian marginalization. We also propose a related approach that further improves the predictive distribution by marginalizing within basins of attraction.
  arXiv  Detail & Related papers  (2020-02-20T15:13:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.