Harden Deep Neural Networks Against Fault Injections Through Weight Scaling

• URL: http://arxiv.org/abs/2411.18993v1
• Date: Thu, 28 Nov 2024 08:47:23 GMT
• Title: Harden Deep Neural Networks Against Fault Injections Through Weight Scaling
• Authors: Ninnart Fuengfusin, Hakaru Tamukoh,
• Abstract summary: We propose a method to harden DNN weights by multiplying weights by constants before storing them to fault-prone medium.<n>Our method is based on the observation that errors from bit-flips have properties similar to additive noise.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks (DNNs) have enabled smart applications on hardware devices. However, these hardware devices are vulnerable to unintended faults caused by aging, temperature variance, and write errors. These faults can cause bit-flips in DNN weights and significantly degrade the performance of DNNs. Thus, protection against these faults is crucial for the deployment of DNNs in critical applications. Previous works have proposed error correction codes based methods, however these methods often require high overheads in both memory and computation. In this paper, we propose a simple yet effective method to harden DNN weights by multiplying weights by constants before storing them to fault-prone medium. When used, these weights are divided back by the same constants to restore the original scale. Our method is based on the observation that errors from bit-flips have properties similar to additive noise, therefore by dividing by constants can reduce the absolute error from bit-flips. To demonstrate our method, we conduct experiments across four ImageNet 2012 pre-trained models along with three different data types: 32-bit floating point, 16-bit floating point, and 8-bit fixed point. This method demonstrates that by only multiplying weights with constants, Top-1 Accuracy of 8-bit fixed point ResNet50 is improved by 54.418 at bit-error rate of 0.0001.

Related papers

• No Data, No Optimization: A Lightweight Method To Disrupt Neural Networks With Sign-Flips [17.136832159667204]
  Deep Neural Lesion (DNL) is a data-free, lightweight method that locates critical parameters and triggers massive accuracy drops. We validate its efficacy on a wide variety of computer vision models and datasets.
  arXiv  Detail & Related papers  (2025-02-11T09:40:45Z)
• Enhancing Neural Network Robustness Against Fault Injection Through Non-linear Weight Transformations [0.0]
  This work focuses on constraining DNN weights by applying saturated activation functions (SAFs)<n>SAFs prevent faults from causing DNN weights to become excessively large, which can lead to model failure.<n>We show that our method enables FP32 ResNet18 with ImageNet 2012 to operate at a bit-error rate of 0.00001 with minor accuracy loss.
  arXiv  Detail & Related papers  (2024-11-28T10:01:39Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• DyBit: Dynamic Bit-Precision Numbers for Efficient Quantized Neural Network Inference [28.912023025671868]
  This work targets an adaptive data representation with variable-length encoding called DyBit. We also propose a hardware-aware quantization framework with a mixed-precision accelerator to trade-off the inference accuracy and speedup. Experimental results demonstrate that the inference accuracy via DyBit is 1.997% higher than the state-of-the-art at 4-bit quantization.
  arXiv  Detail & Related papers  (2023-02-24T08:46:01Z)
• Suppressing quantum errors by scaling a surface code logical qubit [147.2624260358795]
  We report the measurement of logical qubit performance scaling across multiple code sizes. Our system of superconducting qubits has sufficient performance to overcome the additional errors from increasing qubit number. Results mark the first experimental demonstration where quantum error correction begins to improve performance with increasing qubit number.
  arXiv  Detail & Related papers  (2022-07-13T18:00:02Z)
• All-You-Can-Fit 8-Bit Flexible Floating-Point Format for Accurate and Memory-Efficient Inference of Deep Neural Networks [2.294014185517203]
  This paper introduces an extremely flexible 8-bit floating-point (FFP8) format. It achieves an extremely low accuracy loss of $0.1%sim 0.3%$ for several representative image classification models. It is easy to turn a classical floating-point processing unit into an FFP8-compliant one, and the extra hardware cost is minor.
  arXiv  Detail & Related papers  (2021-04-15T09:37:23Z)
• ReCU: Reviving the Dead Weights in Binary Neural Networks [153.6789340484509]
  We explore the influence of "dead weights" which refer to a group of weights that are barely updated during the training of BNNs. We prove that reviving the "dead weights" by ReCU can result in a smaller quantization error. Our method offers not only faster BNN training, but also state-of-the-art performance on CIFAR-10 and ImageNet.
  arXiv  Detail & Related papers  (2021-03-23T08:11:20Z)
• Exponential suppression of bit or phase flip errors with repetitive error correction [56.362599585843085]
  State-of-the-art quantum platforms typically have physical error rates near $10-3$. Quantum error correction (QEC) promises to bridge this divide by distributing quantum logical information across many physical qubits. We implement 1D repetition codes embedded in a 2D grid of superconducting qubits which demonstrate exponential suppression of bit or phase-flip errors.
  arXiv  Detail & Related papers  (2021-02-11T17:11:20Z)
• Bit Error Robustness for Energy-Efficient DNN Accelerators [93.58572811484022]
  We show that a combination of robust fixed-point quantization, weight clipping, and random bit error training (RandBET) improves robustness against random bit errors. This leads to high energy savings from both low-voltage operation as well as low-precision quantization.
  arXiv  Detail & Related papers  (2020-06-24T18:23:10Z)
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks [87.01739569518513]
  We replace conventional ReLU with Bounded ReLU and find that the decline is due to activation quantization. Our integer networks achieve equivalent performance as the corresponding FPN networks, but have only 1/4 memory cost and run 2x faster on modern GPU.
  arXiv  Detail & Related papers  (2020-06-21T08:23:03Z)
• SNIFF: Reverse Engineering of Neural Networks with Fault Attacks [26.542434084399265]
  We explore the possibility to reverse engineer neural networks with the usage of fault attacks. SNIFF stands for sign bit flip fault, which enables the reverse engineering by changing the sign of intermediate values. We develop the first exact extraction method on deep-layer feature extractor networks that provably allows the recovery of the model parameters.
  arXiv  Detail & Related papers  (2020-02-23T05:39:54Z)
• Shifted and Squeezed 8-bit Floating Point format for Low-Precision Training of Deep Neural Networks [13.929168096016957]
  We introduce a novel methodology for training deep neural networks using 8-bit floating point (FP8) numbers. Reduced bit precision allows for a larger effective memory and increased computational speed. We show that, unlike previous 8-bit precision training methods, the proposed method works out-of-the-box for representative models.
  arXiv  Detail & Related papers  (2020-01-16T06:38:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.