Memory Faults in Activation-sparse Quantized Deep Neural Networks: Analysis and Mitigation using Sharpness-aware Training

• URL: http://arxiv.org/abs/2406.10528v1
• Date: Sat, 15 Jun 2024 06:40:48 GMT
• Title: Memory Faults in Activation-sparse Quantized Deep Neural Networks: Analysis and Mitigation using Sharpness-aware Training
• Authors: Akul Malhotra, Sumeet Kumar Gupta,
• Abstract summary: We investigate the impact of memory faults on activation-sparse quantized DNNs (AS QDNNs) We show that a high level of activation sparsity comes at the cost of larger vulnerability to faults, with AS QDNNs exhibiting up to 11.13% lower accuracy than the standard QDNNs. We employ sharpness-aware quantization (SAQ) training to mitigate the impact of memory faults.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Improving the hardware efficiency of deep neural network (DNN) accelerators with techniques such as quantization and sparsity enhancement have shown an immense promise. However, their inference accuracy in non-ideal real-world settings (such as in the presence of hardware faults) is yet to be systematically analyzed. In this work, we investigate the impact of memory faults on activation-sparse quantized DNNs (AS QDNNs). We show that a high level of activation sparsity comes at the cost of larger vulnerability to faults, with AS QDNNs exhibiting up to 11.13% lower accuracy than the standard QDNNs. We establish that the degraded accuracy correlates with a sharper minima in the loss landscape for AS QDNNs, which makes them more sensitive to perturbations in the weight values due to faults. Based on this observation, we employ sharpness-aware quantization (SAQ) training to mitigate the impact of memory faults. The AS and standard QDNNs trained with SAQ have up to 19.50% and 15.82% higher inference accuracy, respectively compared to their conventionally trained equivalents. Moreover, we show that SAQ-trained AS QDNNs show higher accuracy in faulty settings than standard QDNNs trained conventionally. Thus, sharpness-aware training can be instrumental in achieving sparsity-related latency benefits without compromising on fault tolerance.

Related papers

• RescueSNN: Enabling Reliable Executions on Spiking Neural Network Accelerators under Permanent Faults [15.115813664357436]
  RescueSNN is a novel methodology to mitigate permanent faults in the compute engine of SNN chips. RescueSNN improves accuracy by up to 80% while maintaining the throughput reduction below 25% in high fault rate.
  arXiv  Detail & Related papers  (2023-04-08T15:24:57Z)
• Bridging Precision and Confidence: A Train-Time Loss for Calibrating Object Detection [58.789823426981044]
  We propose a novel auxiliary loss formulation that aims to align the class confidence of bounding boxes with the accurateness of predictions. Our results reveal that our train-time loss surpasses strong calibration baselines in reducing calibration error for both in and out-domain scenarios.
  arXiv  Detail & Related papers  (2023-03-25T08:56:21Z)
• FlatENN: Train Flat for Enhanced Fault Tolerance of Quantized Deep Neural Networks [0.03807314298073299]
  We investigate the impact of bit-flip and stuck-at faults on activation-sparse quantized DNNs (QDNNs) We show that a high level of activation sparsity comes at the cost of larger vulnerability to faults. We propose the mitigation of the impact of faults by employing a sharpness-aware quantization scheme.
  arXiv  Detail & Related papers  (2022-12-29T06:06:14Z)
• Fault-Aware Design and Training to Enhance DNNs Reliability with Zero-Overhead [67.87678914831477]
  Deep Neural Networks (DNNs) enable a wide series of technological advancements. Recent findings indicate that transient hardware faults may corrupt the models prediction dramatically. In this work, we propose to tackle the reliability issue both at training and model design time.
  arXiv  Detail & Related papers  (2022-05-28T13:09:30Z)
• HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noise [13.904091056365765]
  We present an SNN training algorithm that uses crafted input noise and incurs no additional training time. Compared to standard trained direct input SNNs, our trained models yield improved classification accuracy of up to 13.7%. Our models also outperform inherently robust SNNs trained on rate-coded inputs with improved or similar classification performance on attack-generated images.
  arXiv  Detail & Related papers  (2021-10-06T16:48:48Z)
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration [74.5509794733707]
  We present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.515% on BNNs. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods.
  arXiv  Detail & Related papers  (2021-02-17T18:59:28Z)
• FAT: Training Neural Networks for Reliable Inference Under Hardware Faults [3.191587417198382]
  We present a novel methodology called fault-aware training (FAT), which includes error modeling during neural network (NN) training, to make QNNs resilient to specific fault models on the device. FAT has been validated for numerous classification tasks including CIFAR10, GTSRB, SVHN and ImageNet.
  arXiv  Detail & Related papers  (2020-11-11T16:09:39Z)
• FATNN: Fast and Accurate Ternary Neural Networks [89.07796377047619]
  Ternary Neural Networks (TNNs) have received much attention due to being potentially orders of magnitude faster in inference, as well as more power efficient, than full-precision counterparts. In this work, we show that, under some mild constraints, computational complexity of the ternary inner product can be reduced by a factor of 2. We elaborately design an implementation-dependent ternary quantization algorithm to mitigate the performance gap.
  arXiv  Detail & Related papers  (2020-08-12T04:26:18Z)
• Bit Error Robustness for Energy-Efficient DNN Accelerators [93.58572811484022]
  We show that a combination of robust fixed-point quantization, weight clipping, and random bit error training (RandBET) improves robustness against random bit errors. This leads to high energy savings from both low-voltage operation as well as low-precision quantization.
  arXiv  Detail & Related papers  (2020-06-24T18:23:10Z)
• On Calibration of Mixup Training for Deep Neural Networks [1.6242924916178283]
  We argue and provide empirical evidence that, due to its fundamentals, Mixup does not necessarily improve calibration. Our loss is inspired by Bayes decision theory and introduces a new training framework for designing losses for probabilistic modelling. We provide state-of-the-art accuracy with consistent improvements in calibration performance.
  arXiv  Detail & Related papers  (2020-03-22T16:54:31Z)
• Widening and Squeezing: Towards Accurate and Efficient QNNs [125.172220129257]
  Quantization neural networks (QNNs) are very attractive to the industry because their extremely cheap calculation and storage overhead, but their performance is still worse than that of networks with full-precision parameters. Most of existing methods aim to enhance performance of QNNs especially binary neural networks by exploiting more effective training techniques. We address this problem by projecting features in original full-precision networks to high-dimensional quantization features.
  arXiv  Detail & Related papers  (2020-02-03T04:11:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.