Impact of Learning Rate on Noise Resistant Property of Deep Learning Models

• URL: http://arxiv.org/abs/2205.07856v1
• Date: Sun, 8 May 2022 00:16:09 GMT
• Title: Impact of Learning Rate on Noise Resistant Property of Deep Learning Models
• Authors: Omobayode Fagbohungbe and Lijun Qian
• Abstract summary: The study is achieved by first training deep learning models using different learning rates. The noise-resistant property of the resulting models is examined by measuring the performance degradation due to the analog noise. The results showed there exists a sweet spot of learning rate values that achieves a good balance between model prediction performance and model noise-resistant property.
• Score: 3.520496620951778
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The interest in analog computation has grown tremendously in recent years due to its fast computation speed and excellent energy efficiency, which is very important for edge and IoT devices in the sub-watt power envelope for deep learning inferencing. However, significant performance degradation suffered by deep learning models due to the inherent noise present in the analog computation can limit their use in mission-critical applications. Hence, there is a need to understand the impact of critical model hyperparameters choice on the resulting model noise-resistant property. This need is critical as the insight obtained can be used to design deep learning models that are robust to analog noise. In this paper, the impact of the learning rate, a critical design choice, on the noise-resistant property is investigated. The study is achieved by first training deep learning models using different learning rates. Thereafter, the models are injected with analog noise and the noise-resistant property of the resulting models is examined by measuring the performance degradation due to the analog noise. The results showed there exists a sweet spot of learning rate values that achieves a good balance between model prediction performance and model noise-resistant property. Furthermore, the theoretical justification of the observed phenomenon is provided.

Related papers

• Towards Robust Transcription: Exploring Noise Injection Strategies for Training Data Augmentation [55.752737615873464]
  This study investigates the impact of white noise at various Signal-to-Noise Ratio (SNR) levels on state-of-the-art APT models. We hope this research provides valuable insights as preliminary work toward developing transcription models that maintain consistent performance across a range of acoustic conditions.
  arXiv  Detail & Related papers  (2024-10-18T02:31:36Z)
• Learning with Noisy Foundation Models [95.50968225050012]
  This paper is the first work to comprehensively understand and analyze the nature of noise in pre-training datasets. We propose a tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise and improve generalization.
  arXiv  Detail & Related papers  (2024-03-11T16:22:41Z)
• A PAC-Bayesian Perspective on the Interpolating Information Criterion [54.548058449535155]
  We show how a PAC-Bayes bound is obtained for a general class of models, characterizing factors which influence performance in the interpolating regime. We quantify how the test error for overparameterized models achieving effectively zero training error depends on the quality of the implicit regularization imposed by e.g. the combination of model, parameter-initialization scheme.
  arXiv  Detail & Related papers  (2023-11-13T01:48:08Z)
• Comparative Study on the Effects of Noise in ML-Based Anxiety Detection [0.0]
  We study how noise impacts model performance and developing models that are robust to noisy, real-world conditions. We compare the effect of various intensities of noise on machine learning models classifying levels of physiological arousal.
  arXiv  Detail & Related papers  (2023-06-01T19:52:24Z)
• Walking Noise: On Layer-Specific Robustness of Neural Architectures against Noisy Computations and Associated Characteristic Learning Dynamics [1.5184189132709105]
  We discuss the implications of additive, multiplicative and mixed noise for different classification tasks and model architectures. We propose a methodology called Walking Noise which injects layer-specific noise to measure the robustness. We conclude with a discussion of the use of this methodology in practice, among others, discussing its use for tailored multi-execution in noisy environments.
  arXiv  Detail & Related papers  (2022-12-20T17:09:08Z)
• Improving the Robustness of Summarization Models by Detecting and Removing Input Noise [50.27105057899601]
  We present a large empirical study quantifying the sometimes severe loss in performance from different types of input noise for a range of datasets and model sizes. We propose a light-weight method for detecting and removing such noise in the input during model inference without requiring any training, auxiliary models, or even prior knowledge of the type of noise.
  arXiv  Detail & Related papers  (2022-12-20T00:33:11Z)
• A Study of Deep CNN Model with Labeling Noise Based on Granular-ball Computing [0.0]
  Granular ball computing is an efficient, robust and scalable learning method. In this paper, we pioneered a granular ball neural network algorithm model, which adopts the idea of multi-granular to filter label noise samples during model training.
  arXiv  Detail & Related papers  (2022-07-17T13:58:46Z)
• Effect of Batch Normalization on Noise Resistant Property of Deep Learning Models [3.520496620951778]
  There are concerns about the presence of analog noise which causes changes to the weight of the models, leading to performance degradation of deep learning model. The effect of the popular batch normalization layer on the noise resistant ability of deep learning model is investigated in this work.
  arXiv  Detail & Related papers  (2022-05-15T20:10:21Z)
• Deep Bayesian Active Learning for Accelerating Stochastic Simulation [74.58219903138301]
  Interactive Neural Process (INP) is a deep active learning framework for simulations and with active learning approaches. For active learning, we propose a novel acquisition function, Latent Information Gain (LIG), calculated in the latent space of NP based models. The results demonstrate STNP outperforms the baselines in the learning setting and LIG achieves the state-of-the-art for active learning.
  arXiv  Detail & Related papers  (2021-06-05T01:31:51Z)
• Benchmarking Inference Performance of Deep Learning Models on Analog Devices [3.520496620951778]
  It is observed that deeper models and models with more redundancy in design such as VGG are more robust to the noise in general. The performance is also affected by the design philosophy of the model, the detailed structure of the model, the exact machine learning task, as well as the datasets.
  arXiv  Detail & Related papers  (2020-11-24T02:14:39Z)
• From Sound Representation to Model Robustness [82.21746840893658]
  We investigate the impact of different standard environmental sound representations (spectrograms) on the recognition performance and adversarial attack robustness of a victim residual convolutional neural network. Averaged over various experiments on three environmental sound datasets, we found the ResNet-18 model outperforms other deep learning architectures.
  arXiv  Detail & Related papers  (2020-07-27T17:30:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.