Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models

• URL: http://arxiv.org/abs/2205.07372v1
• Date: Sun, 15 May 2022 20:10:21 GMT
• Title: Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models
• Authors: Omobayode Fagbohungbe and Lijun Qian
• Abstract summary: There are concerns about the presence of analog noise which causes changes to the weight of the models, leading to performance degradation of deep learning model. The effect of the popular batch normalization layer on the noise resistant ability of deep learning model is investigated in this work.
• Score: 3.520496620951778
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The fast execution speed and energy efficiency of analog hardware has made them a strong contender for deployment of deep learning model at the edge. However, there are concerns about the presence of analog noise which causes changes to the weight of the models, leading to performance degradation of deep learning model, despite their inherent noise resistant characteristics. The effect of the popular batch normalization layer on the noise resistant ability of deep learning model is investigated in this work. This systematic study has been carried out by first training different models with and without batch normalization layer on CIFAR10 and CIFAR100 dataset. The weights of the resulting models are then injected with analog noise and the performance of the models on the test dataset is obtained and compared. The results show that the presence of batch normalization layer negatively impacts noise resistant property of deep learning model and the impact grows with the increase of the number of batch normalization layers.

Related papers

• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• Not All Steps are Equal: Efficient Generation with Progressive Diffusion Models [62.155612146799314]
  We propose a novel two-stage training strategy termed Step-Adaptive Training. In the initial stage, a base denoising model is trained to encompass all timesteps. We partition the timesteps into distinct groups, fine-tuning the model within each group to achieve specialized denoising capabilities.
  arXiv  Detail & Related papers  (2023-12-20T03:32:58Z)
• ExposureDiffusion: Learning to Expose for Low-light Image Enhancement [87.08496758469835]
  This work addresses the issue by seamlessly integrating a diffusion model with a physics-based exposure model. Our method obtains significantly improved performance and reduced inference time compared with vanilla diffusion models. The proposed framework can work with both real-paired datasets, SOTA noise models, and different backbone networks.
  arXiv  Detail & Related papers  (2023-07-15T04:48:35Z)
• Comparative Study on the Effects of Noise in ML-Based Anxiety Detection [0.0]
  We study how noise impacts model performance and developing models that are robust to noisy, real-world conditions. We compare the effect of various intensities of noise on machine learning models classifying levels of physiological arousal.
  arXiv  Detail & Related papers  (2023-06-01T19:52:24Z)
• An Investigation of Noise in Morphological Inflection [21.411766936034]
  We investigate the types of noise encountered within a pipeline for truly unsupervised morphological paradigm completion. We compare the effect of different types of noise on multiple state-of-the-art inflection models. We propose a novel character-level masked language modeling (CMLM) pretraining objective and explore its impact on the models' resistance to noise.
  arXiv  Detail & Related papers  (2023-05-26T02:14:34Z)
• DiffusionAD: Norm-guided One-step Denoising Diffusion for Anomaly Detection [89.49600182243306]
  We reformulate the reconstruction process using a diffusion model into a noise-to-norm paradigm. We propose a rapid one-step denoising paradigm, significantly faster than the traditional iterative denoising in diffusion models. The segmentation sub-network predicts pixel-level anomaly scores using the input image and its anomaly-free restoration.
  arXiv  Detail & Related papers  (2023-03-15T16:14:06Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)
• Impact of Learning Rate on Noise Resistant Property of Deep Learning Models [3.520496620951778]
  The study is achieved by first training deep learning models using different learning rates. The noise-resistant property of the resulting models is examined by measuring the performance degradation due to the analog noise. The results showed there exists a sweet spot of learning rate values that achieves a good balance between model prediction performance and model noise-resistant property.
  arXiv  Detail & Related papers  (2022-05-08T00:16:09Z)
• Treatment Learning Causal Transformer for Noisy Image Classification [62.639851972495094]
  In this work, we incorporate this binary information of "existence of noise" as treatment into image classification tasks to improve prediction accuracy. Motivated from causal variational inference, we propose a transformer-based architecture, that uses a latent generative model to estimate robust feature representations for noise image classification. We also create new noisy image datasets incorporating a wide range of noise factors for performance benchmarking.
  arXiv  Detail & Related papers  (2022-03-29T13:07:53Z)
• Benchmarking Inference Performance of Deep Learning Models on Analog Devices [3.520496620951778]
  It is observed that deeper models and models with more redundancy in design such as VGG are more robust to the noise in general. The performance is also affected by the design philosophy of the model, the detailed structure of the model, the exact machine learning task, as well as the datasets.
  arXiv  Detail & Related papers  (2020-11-24T02:14:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.