Network insensitivity to parameter noise via adversarial regularization

• URL: http://arxiv.org/abs/2106.05009v1
• Date: Wed, 9 Jun 2021 12:11:55 GMT
• Title: Network insensitivity to parameter noise via adversarial regularization
• Authors: Julian B\"ucher, Fynn Faber, Dylan R. Muir
• Abstract summary: We present a new adversarial network optimisation algorithm that attacks network parameters during training. We show that our approach produces models that are more robust to targeted parameter variation. Our work provides an approach to deploy neural network architectures to inference devices that suffer from computational non-idealities.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Neuromorphic neural network processors, in the form of compute-in-memory crossbar arrays of memristors, or in the form of subthreshold analog and mixed-signal ASICs, promise enormous advantages in compute density and energy efficiency for NN-based ML tasks. However, these technologies are prone to computational non-idealities, due to process variation and intrinsic device physics. This degrades the task performance of networks deployed to the processor, by introducing parameter noise into the deployed model. While it is possible to calibrate each device, or train networks individually for each processor, these approaches are expensive and impractical for commercial deployment. Alternative methods are therefore needed to train networks that are inherently robust against parameter variation, as a consequence of network architecture and parameters. We present a new adversarial network optimisation algorithm that attacks network parameters during training, and promotes robust performance during inference in the face of parameter variation. Our approach introduces a regularization term penalising the susceptibility of a network to weight perturbation. We compare against previous approaches for producing parameter insensitivity such as dropout, weight smoothing and introducing parameter noise during training. We show that our approach produces models that are more robust to targeted parameter variation, and equally robust to random parameter variation. Our approach finds minima in flatter locations in the weight-loss landscape compared with other approaches, highlighting that the networks found by our technique are less sensitive to parameter perturbation. Our work provides an approach to deploy neural network architectures to inference devices that suffer from computational non-idealities, with minimal loss of performance. ...

Related papers

• Complexity-Aware Training of Deep Neural Networks for Optimal Structure Discovery [0.0]
  We propose a novel algorithm for combined unit/filter and layer pruning of deep neural networks that functions during training and without requiring a pre-trained network to apply. Our algorithm optimally trades-off learning accuracy and pruning levels while balancing layer vs. unit/filter pruning and computational vs. parameter complexity using only three user-defined parameters.
  arXiv  Detail & Related papers  (2024-11-14T02:00:22Z)
• Task-Oriented Real-time Visual Inference for IoVT Systems: A Co-design Framework of Neural Networks and Edge Deployment [61.20689382879937]
  Task-oriented edge computing addresses this by shifting data analysis to the edge. Existing methods struggle to balance high model performance with low resource consumption. We propose a novel co-design framework to optimize neural network architecture.
  arXiv  Detail & Related papers  (2024-10-29T19:02:54Z)
• Pruning By Explaining Revisited: Optimizing Attribution Methods to Prune CNNs and Transformers [14.756988176469365]
  An effective approach to reduce computational requirements and increase efficiency is to prune unnecessary components of Deep Neural Networks. Previous work has shown that attribution methods from the field of eXplainable AI serve as effective means to extract and prune the least relevant network components in a few-shot fashion.
  arXiv  Detail & Related papers  (2024-08-22T17:35:18Z)
• Rewarded meta-pruning: Meta Learning with Rewards for Channel Pruning [19.978542231976636]
  This paper proposes a novel method to reduce the parameters and FLOPs for computational efficiency in deep learning models. We introduce accuracy and efficiency coefficients to control the trade-off between the accuracy of the network and its computing efficiency.
  arXiv  Detail & Related papers  (2023-01-26T12:32:01Z)
• Learning to Learn with Generative Models of Neural Network Checkpoints [71.06722933442956]
  We construct a dataset of neural network checkpoints and train a generative model on the parameters. We find that our approach successfully generates parameters for a wide range of loss prompts. We apply our method to different neural network architectures and tasks in supervised and reinforcement learning.
  arXiv  Detail & Related papers  (2022-09-26T17:59:58Z)
• An Adaptive Device-Edge Co-Inference Framework Based on Soft Actor-Critic [72.35307086274912]
  High-dimension parameter model and large-scale mathematical calculation restrict execution efficiency, especially for Internet of Things (IoT) devices. We propose a new Deep Reinforcement Learning (DRL)-Soft Actor Critic for discrete (SAC-d), which generates the emphexit point, emphexit point, and emphcompressing bits by soft policy iterations. Based on the latency and accuracy aware reward design, such an computation can well adapt to the complex environment like dynamic wireless channel and arbitrary processing, and is capable of supporting the 5G URL
  arXiv  Detail & Related papers  (2022-01-09T09:31:50Z)
• Rate Distortion Characteristic Modeling for Neural Image Compression [59.25700168404325]
  End-to-end optimization capability offers neural image compression (NIC) superior lossy compression performance. distinct models are required to be trained to reach different points in the rate-distortion (R-D) space. We make efforts to formulate the essential mathematical functions to describe the R-D behavior of NIC using deep network and statistical modeling.
  arXiv  Detail & Related papers  (2021-06-24T12:23:05Z)
• Reduced-Order Neural Network Synthesis with Robustness Guarantees [0.0]
  Machine learning algorithms are being adapted to run locally on board, potentially hardware limited, devices to improve user privacy, reduce latency and be more energy efficient. To address this issue, a method to automatically synthesize reduced-order neural networks (having fewer neurons) approxing the input/output mapping of a larger one is introduced. Worst-case bounds for this approximation error are obtained and the approach can be applied to a wide variety of neural networks architectures.
  arXiv  Detail & Related papers  (2021-02-18T12:03:57Z)
• Supervised training of spiking neural networks for robust deployment on mixed-signal neuromorphic processors [2.6949002029513167]
  Mixed-signal analog/digital electronic circuits can emulate spiking neurons and synapses with extremely high energy efficiency. Mismatch is expressed as differences in effective parameters between identically-configured neurons and synapses. We present a supervised learning approach that addresses this challenge by maximizing robustness to mismatch and other common sources of noise.
  arXiv  Detail & Related papers  (2021-02-12T09:20:49Z)
• Adaptive Subcarrier, Parameter, and Power Allocation for Partitioned Edge Learning Over Broadband Channels [69.18343801164741]
  partitioned edge learning (PARTEL) implements parameter-server training, a well known distributed learning method, in wireless network. We consider the case of deep neural network (DNN) models which can be trained using PARTEL by introducing some auxiliary variables.
  arXiv  Detail & Related papers  (2020-10-08T15:27:50Z)
• Neural Parameter Allocation Search [57.190693718951316]
  Training neural networks requires increasing amounts of memory. Existing methods assume networks have many identical layers and utilize hand-crafted sharing strategies that fail to generalize. We introduce Neural Allocation Search (NPAS), a novel task where the goal is to train a neural network given an arbitrary, fixed parameter budget. NPAS covers both low-budget regimes, which produce compact networks, as well as a novel high-budget regime, where additional capacity can be added to boost performance without increasing inference FLOPs.
  arXiv  Detail & Related papers  (2020-06-18T15:01:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.