Walking Noise: On Layer-Specific Robustness of Neural Architectures against Noisy Computations and Associated Characteristic Learning Dynamics

• URL: http://arxiv.org/abs/2212.10430v2
• Date: Fri, 14 Jun 2024 13:04:54 GMT
• Title: Walking Noise: On Layer-Specific Robustness of Neural Architectures against Noisy Computations and Associated Characteristic Learning Dynamics
• Authors: Hendrik Borras, Bernhard Klein, Holger Fröning,
• Abstract summary: We discuss the implications of additive, multiplicative and mixed noise for different classification tasks and model architectures. We propose a methodology called Walking Noise which injects layer-specific noise to measure the robustness. We conclude with a discussion of the use of this methodology in practice, among others, discussing its use for tailored multi-execution in noisy environments.
• Score: 1.5184189132709105
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep neural networks are extremely successful in various applications, however they exhibit high computational demands and energy consumption. This is exacerbated by stuttering technology scaling, prompting the need for novel approaches to handle increasingly complex neural architectures. At the same time, alternative computing technologies such as analog computing, which promise groundbreaking improvements in energy efficiency, are inevitably fraught with noise and inaccurate calculations. Such noisy computations are more energy efficient, and, given a fixed power budget, also more time efficient. However, like any kind of unsafe optimization, they require countermeasures to ensure functionally correct results. This work considers noisy computations in an abstract form, and gears to understand the implications of such noise on the accuracy of neural network classifiers as an exemplary workload. We propose a methodology called Walking Noise which injects layer-specific noise to measure the robustness and to provide insights on the learning dynamics. In more detail, we investigate the implications of additive, multiplicative and mixed noise for different classification tasks and model architectures. While noisy training significantly increases robustness for all noise types, we observe in particular that it results in increased weight magnitudes and thus inherently improves the signal-to-noise ratio for additive noise injection. Contrarily, training with multiplicative noise can lead to a form of self-binarization of the model parameters, leading to extreme robustness. We conclude with a discussion of the use of this methodology in practice, among others, discussing its use for tailored multi-execution in noisy environments.

Related papers

• Efficient Noise Mitigation for Enhancing Inference Accuracy in DNNs on Mixed-Signal Accelerators [4.416800723562206]
  We model process-induced and aging-related variations of analog computing components on the accuracy of the analog neural networks. We introduce a denoising block inserted between selected layers of a pre-trained model. We demonstrate that training the denoising block significantly increases the model's robustness against various noise levels.
  arXiv  Detail & Related papers  (2024-09-27T08:45:55Z)
• Pivotal Auto-Encoder via Self-Normalizing ReLU [20.76999663290342]
  We formalize single hidden layer sparse auto-encoders as a transform learning problem. We propose an optimization problem that leads to a predictive model invariant to the noise level at test time. Our experimental results demonstrate that the trained models yield a significant improvement in stability against varying types of noise.
  arXiv  Detail & Related papers  (2024-06-23T09:06:52Z)
• Learning Provably Robust Estimators for Inverse Problems via Jittering [51.467236126126366]
  We investigate whether jittering, a simple regularization technique, is effective for learning worst-case robust estimators for inverse problems. We show that jittering significantly enhances the worst-case robustness, but can be suboptimal for inverse problems beyond denoising.
  arXiv  Detail & Related papers  (2023-07-24T14:19:36Z)
• Improve Noise Tolerance of Robust Loss via Noise-Awareness [60.34670515595074]
  We propose a meta-learning method which is capable of adaptively learning a hyper parameter prediction function, called Noise-Aware-Robust-Loss-Adjuster (NARL-Adjuster for brevity) Four SOTA robust loss functions are attempted to be integrated with our algorithm, and comprehensive experiments substantiate the general availability and effectiveness of the proposed method in both its noise tolerance and performance.
  arXiv  Detail & Related papers  (2023-01-18T04:54:58Z)
• Robustness of quantum reinforcement learning under hardware errors [0.0]
  Variational quantum machine learning algorithms have become the focus of recent research on how to utilize near-term quantum devices for machine learning tasks. They are considered suitable for this as the circuits that are run can be tailored to the device, and a big part of the computation is delegated to the classical. However, the effect of training quantum machine learning models under the influence of hardware-induced noise has not yet been extensively studied.
  arXiv  Detail & Related papers  (2022-12-19T13:14:22Z)
• Robust Semantic Communications with Masked VQ-VAE Enabled Codebook [56.63571713657059]
  We propose a framework for the robust end-to-end semantic communication systems to combat the semantic noise. To combat the semantic noise, the adversarial training with weight is developed to incorporate the samples with semantic noise in the training dataset. We develop a feature importance module (FIM) to suppress the noise-related and task-unrelated features.
  arXiv  Detail & Related papers  (2022-06-08T16:58:47Z)
• Characterizing and mitigating coherent errors in a trapped ion quantum processor using hidden inverses [0.20315704654772418]
  Quantum computing testbeds exhibit high-fidelity quantum control over small collections of qubits. These noisy intermediate-scale devices can support a sufficient number of sequential operations prior to decoherence. While the results of these algorithms are imperfect, these imperfections can help bootstrap quantum computer testbed development.
  arXiv  Detail & Related papers  (2022-05-27T20:35:24Z)
• Practical Blind Image Denoising via Swin-Conv-UNet and Data Synthesis [148.16279746287452]
  We propose a swin-conv block to incorporate the local modeling ability of residual convolutional layer and non-local modeling ability of swin transformer block. For the training data synthesis, we design a practical noise degradation model which takes into consideration different kinds of noise. Experiments on AGWN removal and real image denoising demonstrate that the new network architecture design achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-03-24T18:11:31Z)
• Removing Noise from Extracellular Neural Recordings Using Fully Convolutional Denoising Autoencoders [62.997667081978825]
  We propose a Fully Convolutional Denoising Autoencoder, which learns to produce a clean neuronal activity signal from a noisy multichannel input. The experimental results on simulated data show that our proposed method can improve significantly the quality of noise-corrupted neural signals.
  arXiv  Detail & Related papers  (2021-09-18T14:51:24Z)
• Learning based signal detection for MIMO systems with unknown noise statistics [84.02122699723536]
  This paper aims to devise a generalized maximum likelihood (ML) estimator to robustly detect signals with unknown noise statistics. In practice, there is little or even no statistical knowledge on the system noise, which in many cases is non-Gaussian, impulsive and not analyzable. Our framework is driven by an unsupervised learning approach, where only the noise samples are required.
  arXiv  Detail & Related papers  (2021-01-21T04:48:15Z)
• Robust Processing-In-Memory Neural Networks via Noise-Aware Normalization [26.270754571140735]
  PIM accelerators often suffer from intrinsic noise in the physical components. We propose a noise-agnostic method to achieve robust neural network performance against any noise setting.
  arXiv  Detail & Related papers  (2020-07-07T06:51:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.