Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models

• URL: http://arxiv.org/abs/2205.04886v1
• Date: Sat, 7 May 2022 22:23:21 GMT
• Title: Impact of L1 Batch Normalization on Analog Noise Resistant Property of Deep Learning Models
• Authors: Omobayode Fagbohungbe and Lijun Qian
• Abstract summary: In this work, the use of L1 or TopK BatchNorm type in designing deep neural network (DNN) models with excellent noise-resistant property is proposed. The resulting model noise-resistant property is tested by injecting additive noise to the model weights and evaluating the new model inference accuracy due to the noise. The results show that L1 and TopK BatchNorm type has excellent noise-resistant property, and there is no sacrifice in performance due to the change in the BatchNorm type from L2 to L1/TopK BatchNorm type.
• Score: 3.520496620951778
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Analog hardware has become a popular choice for machine learning on resource-constrained devices recently due to its fast execution and energy efficiency. However, the inherent presence of noise in analog hardware and the negative impact of the noise on deployed deep neural network (DNN) models limit their usage. The degradation in performance due to the noise calls for the novel design of DNN models that have excellent noiseresistant property, leveraging the properties of the fundamental building block of DNN models. In this work, the use of L1 or TopK BatchNorm type, a fundamental DNN model building block, in designing DNN models with excellent noise-resistant property is proposed. Specifically, a systematic study has been carried out by training DNN models with L1/TopK BatchNorm type, and the performance is compared with DNN models with L2 BatchNorm types. The resulting model noise-resistant property is tested by injecting additive noise to the model weights and evaluating the new model inference accuracy due to the noise. The results show that L1 and TopK BatchNorm type has excellent noise-resistant property, and there is no sacrifice in performance due to the change in the BatchNorm type from L2 to L1/TopK BatchNorm type.

Related papers

• sVAD: A Robust, Low-Power, and Light-Weight Voice Activity Detection with Spiking Neural Networks [51.516451451719654]
  Spiking Neural Networks (SNNs) are known to be biologically plausible and power-efficient. This paper introduces a novel SNN-based Voice Activity Detection model, referred to as sVAD. It provides effective auditory feature representation through SincNet and 1D convolution, and improves noise robustness with attention mechanisms.
  arXiv  Detail & Related papers  (2024-03-09T02:55:44Z)
• Noise Sensitivity and Stability of Deep Neural Networks for Binary Classification [0.9438207505148947]
  We ask if certain sequences of Boolean functions represented by common DNN models are noise sensitive or noise stable. Due to the natural randomness in DNN models, these concepts are extended to annealed and quenched versions. We investigate the properties of two standard DNN architectures, the fully connected and convolutional models, when initiated with Gaussian weights.
  arXiv  Detail & Related papers  (2023-08-18T08:09:31Z)
• Variational Positive-incentive Noise: How Noise Benefits Models [84.67629229767047]
  We investigate how to benefit the classical models by random noise under the framework of Positive-incentive Noise (Pi-Noise) Since the ideal objective of Pi-Noise is intractable, we propose to optimize its variational bound instead, namely variational Pi-Noise (VPN)
  arXiv  Detail & Related papers  (2023-06-13T09:43:32Z)
• Latent Class-Conditional Noise Model [54.56899309997246]
  We introduce a Latent Class-Conditional Noise model (LCCN) to parameterize the noise transition under a Bayesian framework. We then deduce a dynamic label regression method for LCCN, whose Gibbs sampler allows us efficiently infer the latent true labels. Our approach safeguards the stable update of the noise transition, which avoids previous arbitrarily tuning from a mini-batch of samples.
  arXiv  Detail & Related papers  (2023-02-19T15:24:37Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)
• Towards Robust k-Nearest-Neighbor Machine Translation [72.9252395037097]
  k-Nearest-Neighbor Machine Translation (kNN-MT) becomes an important research direction of NMT in recent years. Its main idea is to retrieve useful key-value pairs from an additional datastore to modify translations without updating the NMT model. The underlying retrieved noisy pairs will dramatically deteriorate the model performance. We propose a confidence-enhanced kNN-MT model with robust training to alleviate the impact of noise.
  arXiv  Detail & Related papers  (2022-10-17T07:43:39Z)
• Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models [3.520496620951778]
  There are concerns about the presence of analog noise which causes changes to the weight of the models, leading to performance degradation of deep learning model. The effect of the popular batch normalization layer on the noise resistant ability of deep learning model is investigated in this work.
  arXiv  Detail & Related papers  (2022-05-15T20:10:21Z)
• Impact of Learning Rate on Noise Resistant Property of Deep Learning Models [3.520496620951778]
  The study is achieved by first training deep learning models using different learning rates. The noise-resistant property of the resulting models is examined by measuring the performance degradation due to the analog noise. The results showed there exists a sweet spot of learning rate values that achieves a good balance between model prediction performance and model noise-resistant property.
  arXiv  Detail & Related papers  (2022-05-08T00:16:09Z)
• C2N: Practical Generative Noise Modeling for Real-World Denoising [53.96391787869974]
  We introduce a Clean-to-Noisy image generation framework, namely C2N, to imitate complex real-world noise without using paired examples. We construct the noise generator in C2N accordingly with each component of real-world noise characteristics to express a wide range of noise accurately.
  arXiv  Detail & Related papers  (2022-02-19T05:53:46Z)
• Denoising Noisy Neural Networks: A Bayesian Approach with Compensation [36.39188653838991]
  Noisy neural networks (NoisyNNs) refer to the inference and training of NNs in the presence of noise. This paper studies how to estimate the uncontaminated NN weights from their noisy observations or manifestations.
  arXiv  Detail & Related papers  (2021-05-22T11:51:20Z)
• Robust Learning of Recurrent Neural Networks in Presence of Exogenous Noise [22.690064709532873]
  We propose a tractable robustness analysis for RNN models subject to input noise. The robustness measure can be estimated efficiently using linearization techniques. Our proposed methodology significantly improves robustness of recurrent neural networks.
  arXiv  Detail & Related papers  (2021-05-03T16:45:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.