Bit Error Tolerance Metrics for Binarized Neural Networks

• URL: http://arxiv.org/abs/2102.01344v1
• Date: Tue, 2 Feb 2021 06:44:55 GMT
• Title: Bit Error Tolerance Metrics for Binarized Neural Networks
• Authors: Sebastian Buschj\"ager, Jian-Jia Chen, Kuan-Hsun Chen, Mario G\"unzel, Katharina Morik, Rodion Novkin, Lukas Pfahler, Mikail Yayla
• Abstract summary: We investigate the internal changes in the neural network (NN) that bit flip training causes, with a focus on binarized NNs (BNNs) We propose a neuron-level bit error tolerance metric, which calculates the margin between the pre-activation values and batch normalization thresholds. We also propose an inter-neuron bit error tolerance metric, which measures the importance of each neuron and computes the variance over all importance values.
• Score: 8.863516255789408
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To reduce the resource demand of neural network (NN) inference systems, it has been proposed to use approximate memory, in which the supply voltage and the timing parameters are tuned trading accuracy with energy consumption and performance. Tuning these parameters aggressively leads to bit errors, which can be tolerated by NNs when bit flips are injected during training. However, bit flip training, which is the state of the art for achieving bit error tolerance, does not scale well; it leads to massive overheads and cannot be applied for high bit error rates (BERs). Alternative methods to achieve bit error tolerance in NNs are needed, but the underlying principles behind the bit error tolerance of NNs have not been reported yet. With this lack of understanding, further progress in the research on NN bit error tolerance will be restrained. In this study, our objective is to investigate the internal changes in the NNs that bit flip training causes, with a focus on binarized NNs (BNNs). To this end, we quantify the properties of bit error tolerant BNNs with two metrics. First, we propose a neuron-level bit error tolerance metric, which calculates the margin between the pre-activation values and batch normalization thresholds. Secondly, to capture the effects of bit error tolerance on the interplay of neurons, we propose an inter-neuron bit error tolerance metric, which measures the importance of each neuron and computes the variance over all importance values. Our experimental results support that these two metrics are strongly related to bit error tolerance.

Related papers

• ZOBNN: Zero-Overhead Dependable Design of Binary Neural Networks with Deliberately Quantized Parameters [0.0]
  In this paper, we introduce a third advantage of very low-precision neural networks: improved fault-tolerance. We investigate the impact of memory faults on state-of-the-art binary neural networks (BNNs) through comprehensive analysis. We propose a technique to improve BNN dependability by restricting the range of float parameters through a novel deliberately uniform quantization.
  arXiv  Detail & Related papers  (2024-07-06T05:31:11Z)
• Guaranteed Approximation Bounds for Mixed-Precision Neural Operators [83.64404557466528]
  We build on intuition that neural operator learning inherently induces an approximation error. We show that our approach reduces GPU memory usage by up to 50% and improves throughput by 58% with little or no reduction in accuracy.
  arXiv  Detail & Related papers  (2023-07-27T17:42:06Z)
• An Estimator for the Sensitivity to Perturbations of Deep Neural Networks [0.31498833540989407]
  This paper derives an estimator that can predict the sensitivity of a given Deep Neural Network to perturbations in input. An approximation of the estimator is tested on two Convolutional Neural Networks, AlexNet and VGG-19, using the ImageNet dataset.
  arXiv  Detail & Related papers  (2023-07-24T10:33:32Z)
• Benign Overfitting in Deep Neural Networks under Lazy Training [72.28294823115502]
  We show that when the data distribution is well-separated, DNNs can achieve Bayes-optimal test error for classification. Our results indicate that interpolating with smoother functions leads to better generalization.
  arXiv  Detail & Related papers  (2023-05-30T19:37:44Z)
• Can pruning improve certified robustness of neural networks? [106.03070538582222]
  We show that neural network pruning can improve empirical robustness of deep neural networks (NNs) Our experiments show that by appropriately pruning an NN, its certified accuracy can be boosted up to 8.2% under standard training. We additionally observe the existence of certified lottery tickets that can match both standard and certified robust accuracies of the original dense models.
  arXiv  Detail & Related papers  (2022-06-15T05:48:51Z)
• Converting Artificial Neural Networks to Spiking Neural Networks via Parameter Calibration [21.117214351356765]
  Spiking Neural Network (SNN) is recognized as one of the next-generation neural networks. In this work, we argue that simply copying and pasting the weights of ANN to SNN inevitably results in activation mismatch. We propose a set of layer-wise parameter calibration algorithms, which adjusts the parameters to minimize the activation mismatch.
  arXiv  Detail & Related papers  (2022-05-06T18:22:09Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Random and Adversarial Bit Error Robustness: Energy-Efficient and Secure DNN Accelerators [105.60654479548356]
  We show that a combination of robust fixed-point quantization, weight clipping, as well as random bit error training (RandBET) improves robustness against random or adversarial bit errors in quantized DNN weights significantly. This leads to high energy savings for low-voltage operation as well as low-precision quantization, but also improves security of DNN accelerators.
  arXiv  Detail & Related papers  (2021-04-16T19:11:14Z)
• Improving Accuracy of Binary Neural Networks using Unbalanced Activation Distribution [12.46127622357824]
  We show that unbalanced activation distribution can actually improve the accuracy of BNNs. We also show that adjusting the threshold values of binary activation functions results in the unbalanced distribution of the binary activation. Experimental results show that the accuracy of previous BNN models can be improved by simply shifting the threshold values of binary activation functions.
  arXiv  Detail & Related papers  (2020-12-02T02:49:53Z)
• Bit Error Robustness for Energy-Efficient DNN Accelerators [93.58572811484022]
  We show that a combination of robust fixed-point quantization, weight clipping, and random bit error training (RandBET) improves robustness against random bit errors. This leads to high energy savings from both low-voltage operation as well as low-precision quantization.
  arXiv  Detail & Related papers  (2020-06-24T18:23:10Z)
• Towards Explainable Bit Error Tolerance of Resistive RAM-Based Binarized Neural Networks [7.349786872131006]
  Non-volatile memory, such as resistive RAM (RRAM), is an emerging energy-efficient storage. Binary neural networks (BNNs) can tolerate a certain percentage of errors without a loss in accuracy. The bit error tolerance (BET) in BNNs can be achieved by flipping the weight signs during training.
  arXiv  Detail & Related papers  (2020-02-03T17:38:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.