Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning

• URL: http://arxiv.org/abs/2407.02721v1
• Date: Wed, 3 Jul 2024 00:25:25 GMT
• Title: Model and Feature Diversity for Bayesian Neural Networks in Mutual Learning
• Authors: Cuong Pham, Cuong C. Nguyen, Trung Le, Dinh Phung, Gustavo Carneiro, Thanh-Toan Do,
• Abstract summary: We propose a novel approach to improve BNNs performance through deep mutual learning. Experimental results demonstrate significant improvements in the classification accuracy, negative log-likelihood, and expected calibration error.
• Score: 33.629630904922465
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bayesian Neural Networks (BNNs) offer probability distributions for model parameters, enabling uncertainty quantification in predictions. However, they often underperform compared to deterministic neural networks. Utilizing mutual learning can effectively enhance the performance of peer BNNs. In this paper, we propose a novel approach to improve BNNs performance through deep mutual learning. The proposed approaches aim to increase diversity in both network parameter distributions and feature distributions, promoting peer networks to acquire distinct features that capture different characteristics of the input, which enhances the effectiveness of mutual learning. Experimental results demonstrate significant improvements in the classification accuracy, negative log-likelihood, and expected calibration error when compared to traditional mutual learning for BNNs.

Related papers

• Edge AI Collaborative Learning: Bayesian Approaches to Uncertainty Estimation [0.0]
  We focus on determining confidence levels in learning outcomes considering the spatial variability of data encountered by independent agents. We implement a 3D environment simulation using the Webots platform to simulate collaborative mapping tasks. Experiments demonstrate that BNNs can effectively support uncertainty estimation in a distributed learning context.
  arXiv  Detail & Related papers  (2024-10-11T09:20:16Z)
• BEND: Bagging Deep Learning Training Based on Efficient Neural Network Diffusion [56.9358325168226]
  We propose a Bagging deep learning training algorithm based on Efficient Neural network Diffusion (BEND) Our approach is simple but effective, first using multiple trained model weights and biases as inputs to train autoencoder and latent diffusion model. Our proposed BEND algorithm can consistently outperform the mean and median accuracies of both the original trained model and the diffused model.
  arXiv  Detail & Related papers  (2024-03-23T08:40:38Z)
• A Framework for Variational Inference of Lightweight Bayesian Neural Networks with Heteroscedastic Uncertainties [0.31457219084519006]
  We show that both the heteroscedastic aleatoric and epistemic variance can be embedded into the variances of learned BNN parameters. We introduce a relatively simple framework for sampling-free variational inference suitable for lightweight BNNs.
  arXiv  Detail & Related papers  (2024-02-22T13:24:43Z)
• Masked Bayesian Neural Networks : Theoretical Guarantee and its Posterior Inference [1.2722697496405464]
  We propose a new node-sparse BNN model which has good theoretical properties and is computationally feasible. We prove that the posterior concentration rate to the true model is near minimax optimal and adaptive to the smoothness of the true model. In addition, we develop a novel MCMC algorithm which makes the Bayesian inference of the node-sparse BNN model feasible in practice.
  arXiv  Detail & Related papers  (2023-05-24T06:16:11Z)
• Efficient Bayes Inference in Neural Networks through Adaptive Importance Sampling [19.518237361775533]
  In BNNs, a complete posterior distribution of the unknown weight and bias parameters of the network is produced during the training stage. This feature is useful in countless machine learning applications. It is particularly appealing in areas where decision-making has a crucial impact, such as medical healthcare or autonomous driving.
  arXiv  Detail & Related papers  (2022-10-03T14:59:23Z)
• Variational Neural Networks [88.24021148516319]
  We propose a method for uncertainty estimation in neural networks called Variational Neural Network (VNN) VNN generates parameters for the output distribution of a layer by transforming its inputs with learnable sub-layers. In uncertainty quality estimation experiments, we show that VNNs achieve better uncertainty quality than Monte Carlo Dropout or Bayes By Backpropagation methods.
  arXiv  Detail & Related papers  (2022-07-04T15:41:02Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layer [77.78479877473899]
  We design a spatial-temporal-fusion BNN for efficiently scaling BNNs to large models. Compared to vanilla BNNs, our approach can greatly reduce the training time and the number of parameters, which contributes to scale BNNs efficiently.
  arXiv  Detail & Related papers  (2021-12-12T17:13:14Z)
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration [74.5509794733707]
  We present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.515% on BNNs. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods.
  arXiv  Detail & Related papers  (2021-02-17T18:59:28Z)
• Network Diffusions via Neural Mean-Field Dynamics [52.091487866968286]
  We propose a novel learning framework for inference and estimation problems of diffusion on networks. Our framework is derived from the Mori-Zwanzig formalism to obtain an exact evolution of the node infection probabilities. Our approach is versatile and robust to variations of the underlying diffusion network models.
  arXiv  Detail & Related papers  (2020-06-16T18:45:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.