Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving

• URL: http://arxiv.org/abs/2110.03553v1
• Date: Thu, 7 Oct 2021 15:20:53 GMT
• Title: Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network Training via Memory-Friendly Pattern Retrieving
• Authors: Qiyu Wan, Haojun Xia, Xingyao Zhang, Lening Wang, Shuaiwen Leon Song, Xin Fu
• Abstract summary: We design and prototype the first highly efficient BNN training accelerator, named Shift-BNN, that is low-cost and scalable. Shift-BNN achieves an average of 4.9x (up to 10.8x) boost in energy efficiency and 1.6x (up to 2.8x) speedup over the baseline DNN training accelerator.
• Score: 5.043640793217879
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Bayesian Neural Networks (BNNs) that possess a property of uncertainty estimation have been increasingly adopted in a wide range of safety-critical AI applications which demand reliable and robust decision making, e.g., self-driving, rescue robots, medical image diagnosis. The training procedure of a probabilistic BNN model involves training an ensemble of sampled DNN models, which induces orders of magnitude larger volume of data movement than training a single DNN model. In this paper, we reveal that the root cause for BNN training inefficiency originates from the massive off-chip data transfer by Gaussian Random Variables (GRVs). To tackle this challenge, we propose a novel design that eliminates all the off-chip data transfer by GRVs through the reversed shifting of Linear Feedback Shift Registers (LFSRs) without incurring any training accuracy loss. To efficiently support our LFSR reversion strategy at the hardware level, we explore the design space of the current DNN accelerators and identify the optimal computation mapping scheme to best accommodate our strategy. By leveraging this finding, we design and prototype the first highly efficient BNN training accelerator, named Shift-BNN, that is low-cost and scalable. Extensive evaluation on five representative BNN models demonstrates that Shift-BNN achieves an average of 4.9x (up to 10.8x) boost in energy efficiency and 1.6x (up to 2.8x) speedup over the baseline DNN training accelerator.

Related papers

• NAS-BNN: Neural Architecture Search for Binary Neural Networks [55.058512316210056]
  We propose a novel neural architecture search scheme for binary neural networks, named NAS-BNN. Our discovered binary model family outperforms previous BNNs for a wide range of operations (OPs) from 20M to 200M. In addition, we validate the transferability of these searched BNNs on the object detection task, and our binary detectors with the searched BNNs achieve a novel state-of-the-art result, e.g., 31.6% mAP with 370M OPs, on MS dataset.
  arXiv  Detail & Related papers  (2024-08-28T02:17:58Z)
• Make Me a BNN: A Simple Strategy for Estimating Bayesian Uncertainty from Pre-trained Models [40.38541033389344]
  Deep Neural Networks (DNNs) are powerful tools for various computer vision tasks, yet they often struggle with reliable uncertainty quantification. We introduce the Adaptable Bayesian Neural Network (ABNN), a simple and scalable strategy to seamlessly transform DNNs into BNNs. We conduct extensive experiments across multiple datasets for image classification and semantic segmentation tasks, and our results demonstrate that ABNN achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-12-23T16:39:24Z)
• Recurrent Bilinear Optimization for Binary Neural Networks [58.972212365275595]
  BNNs neglect the intrinsic bilinear relationship of real-valued weights and scale factors. Our work is the first attempt to optimize BNNs from the bilinear perspective. We obtain robust RBONNs, which show impressive performance over state-of-the-art BNNs on various models and datasets.
  arXiv  Detail & Related papers  (2022-09-04T06:45:33Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Rethinking Pretraining as a Bridge from ANNs to SNNs [13.984523794353477]
  Spiking neural networks (SNNs) are known as a typical kind of brain-inspired models with their unique features. How to obtain a high-accuracy model has always been the main challenge in the field of SNN.
  arXiv  Detail & Related papers  (2022-03-02T14:59:57Z)
• Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layer [77.78479877473899]
  We design a spatial-temporal-fusion BNN for efficiently scaling BNNs to large models. Compared to vanilla BNNs, our approach can greatly reduce the training time and the number of parameters, which contributes to scale BNNs efficiently.
  arXiv  Detail & Related papers  (2021-12-12T17:13:14Z)
• Robustness of Bayesian Neural Networks to White-Box Adversarial Attacks [55.531896312724555]
  Bayesian Networks (BNNs) are robust and adept at handling adversarial attacks by incorporating randomness. We create our BNN model, called BNN-DenseNet, by fusing Bayesian inference (i.e., variational Bayes) to the DenseNet architecture. An adversarially-trained BNN outperforms its non-Bayesian, adversarially-trained counterpart in most experiments.
  arXiv  Detail & Related papers  (2021-11-16T16:14:44Z)
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration [74.5509794733707]
  We present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.515% on BNNs. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods.
  arXiv  Detail & Related papers  (2021-02-17T18:59:28Z)
• FTBNN: Rethinking Non-linearity for 1-bit CNNs and Going Beyond [23.5996182207431]
  We show that binarized convolution process owns an increasing linearity towards the target of minimizing such error, which in turn hampers BNN's discriminative ability. We re-investigate and tune proper non-linear modules to fix that contradiction, leading to a strong baseline which achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-10-19T08:11:48Z)
• TaxoNN: A Light-Weight Accelerator for Deep Neural Network Training [2.5025363034899732]
  We present a novel approach to add the training ability to a baseline DNN accelerator (inference only) by splitting the SGD algorithm into simple computational elements. Based on this approach we propose TaxoNN, a light-weight accelerator for DNN training. Our experimental results show that TaxoNN delivers, on average, 0.97% higher misclassification rate compared to a full-precision implementation.
  arXiv  Detail & Related papers  (2020-10-11T09:04:19Z)
• TxSim:Modeling Training of Deep Neural Networks on Resistive Crossbar Systems [3.1887081453726136]
  crossbar-based computations face a major challenge due to a variety of device and circuit-level non-idealities. We propose TxSim, a fast and customizable modeling framework to functionally evaluate DNN training on crossbar-based hardware.
  arXiv  Detail & Related papers  (2020-02-25T19:29:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.