Variance-Aware Noisy Training: Hardening DNNs against Unstable Analog Computations

• URL: http://arxiv.org/abs/2503.16183v1
• Date: Thu, 20 Mar 2025 14:34:03 GMT
• Title: Variance-Aware Noisy Training: Hardening DNNs against Unstable Analog Computations
• Authors: Xiao Wang, Hendrik Borras, Bernhard Klein, Holger Fröning,
• Abstract summary: Noisy Training is one of the most effective techniques for enhancing robustness, but its performance degrades in real-world environments.<n>We propose Variance-Aware Noisy Training, a novel approach that incorporates noise schedules which emulate the evolving noise conditions encountered during inference.<n>We demonstrate a significant increase in robustness, from 72.3% with conventional Noisy Training to 97.3% with Variance-Aware Noisy Training on CIFAR-10 and from 38.5% to 89.9% on Tiny ImageNet.
• Score: 5.975221928631025
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The disparity between the computational demands of deep learning and the capabilities of compute hardware is expanding drastically. Although deep learning achieves remarkable performance in countless tasks, its escalating requirements for computational power and energy consumption surpass the sustainable limits of even specialized neural processing units, including the Apple Neural Engine and NVIDIA TensorCores. This challenge is intensified by the slowdown in CMOS scaling. Analog computing presents a promising alternative, offering substantial improvements in energy efficiency by directly manipulating physical quantities such as current, voltage, charge, or photons. However, it is inherently vulnerable to manufacturing variations, nonlinearities, and noise, leading to degraded prediction accuracy. One of the most effective techniques for enhancing robustness, Noisy Training, introduces noise during the training phase to reinforce the model against disturbances encountered during inference. Although highly effective, its performance degrades in real-world environments where noise characteristics fluctuate due to external factors such as temperature variations and temporal drift. This study underscores the necessity of Noisy Training while revealing its fundamental limitations in the presence of dynamic noise. To address these challenges, we propose Variance-Aware Noisy Training, a novel approach that mitigates performance degradation by incorporating noise schedules which emulate the evolving noise conditions encountered during inference. Our method substantially improves model robustness, without training overhead. We demonstrate a significant increase in robustness, from 72.3\% with conventional Noisy Training to 97.3\% with Variance-Aware Noisy Training on CIFAR-10 and from 38.5\% to 89.9\% on Tiny ImageNet.

Related papers

• HADL Framework for Noise Resilient Long-Term Time Series Forecasting [0.7810572107832383]
  Long-term time series forecasting is critical in domains such as finance, economics, and energy. The impact of temporal noise in extended lookback windows remains underexplored, often degrading model performance and computational efficiency. We propose a novel framework that addresses these challenges by integrating the Discrete Wavelet Transform (DWT) and Discrete Cosine Transform (DCT) Our approach demonstrates competitive robustness to noisy input, significantly reduces computational complexity, and achieves competitive or state-of-the-art forecasting performance across diverse benchmark datasets.
  arXiv  Detail & Related papers  (2025-02-14T21:41:42Z)
• Towards Robust Transcription: Exploring Noise Injection Strategies for Training Data Augmentation [55.752737615873464]
  This study investigates the impact of white noise at various Signal-to-Noise Ratio (SNR) levels on state-of-the-art APT models. We hope this research provides valuable insights as preliminary work toward developing transcription models that maintain consistent performance across a range of acoustic conditions.
  arXiv  Detail & Related papers  (2024-10-18T02:31:36Z)
• Robust Federated Learning Over the Air: Combating Heavy-Tailed Noise with Median Anchored Clipping [57.40251549664762]
  We propose a novel gradient clipping method, termed Median Anchored Clipping (MAC), to combat the detrimental effects of heavy-tailed noise.<n>We also derive analytical expressions for the convergence rate of model training with analog over-the-air federated learning under MAC.
  arXiv  Detail & Related papers  (2024-09-23T15:11:40Z)
• Improved Noise Schedule for Diffusion Training [51.849746576387375]
  We propose a novel approach to design the noise schedule for enhancing the training of diffusion models. We empirically demonstrate the superiority of our noise schedule over the standard cosine schedule.
  arXiv  Detail & Related papers  (2024-07-03T17:34:55Z)
• High Noise Scheduling is a Must [7.694256285730863]
  Consistency models possess high capabilities for image generation, advancing sampling steps to a single step through their advanced techniques. Current advancements move one step forward consistency training techniques and eliminates the limitation of training. In this study, it is investigated the balance between high and low noise levels in noise distribution and offered noise distribution to maintain the stability.
  arXiv  Detail & Related papers  (2024-04-09T14:44:12Z)
• DOCTOR: Dynamic On-Chip Temporal Variation Remediation Toward Self-Corrected Photonic Tensor Accelerators [5.873308516576125]
  Photonic tensor accelerators offer unparalleled speed and energy efficiency. Off-chip noise-aware training and on-chip training have been proposed to enhance the variation tolerance of optical neural accelerators. We propose a lightweight dynamic on-chip framework, dubbed DOCTOR, providing adaptive, in-situ accuracy recovery against temporally drifting noise.
  arXiv  Detail & Related papers  (2024-03-05T06:17:13Z)
• Adaptive Training Meets Progressive Scaling: Elevating Efficiency in Diffusion Models [52.1809084559048]
  We propose a novel two-stage divide-and-conquer training strategy termed TDC Training.<n>It groups timesteps based on task similarity and difficulty, assigning highly customized denoising models to each group, thereby enhancing the performance of diffusion models.<n>While two-stage training avoids the need to train each model separately, the total training cost is even lower than training a single unified denoising model.
  arXiv  Detail & Related papers  (2023-12-20T03:32:58Z)
• Realistic Noise Synthesis with Diffusion Models [44.404059914652194]
  Deep denoising models require extensive real-world training data, which is challenging to acquire.<n>We propose a novel Realistic Noise Synthesis Diffusor (RNSD) method using diffusion models to address these challenges.
  arXiv  Detail & Related papers  (2023-05-23T12:56:01Z)
• Latent Class-Conditional Noise Model [54.56899309997246]
  We introduce a Latent Class-Conditional Noise model (LCCN) to parameterize the noise transition under a Bayesian framework. We then deduce a dynamic label regression method for LCCN, whose Gibbs sampler allows us efficiently infer the latent true labels. Our approach safeguards the stable update of the noise transition, which avoids previous arbitrarily tuning from a mini-batch of samples.
  arXiv  Detail & Related papers  (2023-02-19T15:24:37Z)
• Improve Noise Tolerance of Robust Loss via Noise-Awareness [60.34670515595074]
  We propose a meta-learning method which is capable of adaptively learning a hyper parameter prediction function, called Noise-Aware-Robust-Loss-Adjuster (NARL-Adjuster for brevity) Four SOTA robust loss functions are attempted to be integrated with our algorithm, and comprehensive experiments substantiate the general availability and effectiveness of the proposed method in both its noise tolerance and performance.
  arXiv  Detail & Related papers  (2023-01-18T04:54:58Z)
• Walking Noise: On Layer-Specific Robustness of Neural Architectures against Noisy Computations and Associated Characteristic Learning Dynamics [1.5184189132709105]
  We discuss the implications of additive, multiplicative and mixed noise for different classification tasks and model architectures. We propose a methodology called Walking Noise which injects layer-specific noise to measure the robustness. We conclude with a discussion of the use of this methodology in practice, among others, discussing its use for tailored multi-execution in noisy environments.
  arXiv  Detail & Related papers  (2022-12-20T17:09:08Z)
• SAMSON: Sharpness-Aware Minimization Scaled by Outlier Normalization for Improving DNN Generalization and Robustness [11.249410336982258]
  Energy-efficient deep neural network (DNN) accelerators are prone to non-idealities that degrade performance at inference time. Existing methods typically add perturbations to the DNN weights during training to simulate inference on noisy hardware. We show that applying sharpness-aware training, by optimizing for both the loss value and loss sharpness, significantly improves robustness to noisy hardware at inference time.
  arXiv  Detail & Related papers  (2022-11-18T16:58:23Z)
• On Dynamic Noise Influence in Differentially Private Learning [102.6791870228147]
  Private Gradient Descent (PGD) is a commonly used private learning framework, which noises based on the Differential protocol. Recent studies show that emphdynamic privacy schedules can improve at the final iteration, yet yet theoreticals of the effectiveness of such schedules remain limited. This paper provides comprehensive analysis of noise influence in dynamic privacy schedules to answer these critical questions.
  arXiv  Detail & Related papers  (2021-01-19T02:04:00Z)
• Robust Processing-In-Memory Neural Networks via Noise-Aware Normalization [26.270754571140735]
  PIM accelerators often suffer from intrinsic noise in the physical components. We propose a noise-agnostic method to achieve robust neural network performance against any noise setting.
  arXiv  Detail & Related papers  (2020-07-07T06:51:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.