Blending Optimal Control and Biologically Plausible Learning for Noise-Robust Physical Neural Networks

• URL: http://arxiv.org/abs/2502.19053v1
• Date: Wed, 26 Feb 2025 11:10:40 GMT
• Title: Blending Optimal Control and Biologically Plausible Learning for Noise-Robust Physical Neural Networks
• Authors: Satoshi Sunada, Tomoaki Niiyama, Kazutaka Kanno, Rin Nogami, André Röhm, Takato Awano, Atsushi Uchida,
• Abstract summary: Physical neural networks (PNNs) offer efficient neuromorphic information processing.<n>We propose a training approach for substantially reducing this training cost.<n>In addition to the reduction of training time, this approach achieves robust processing even under measurement errors and noise.
• Score: 0.9423257767158634
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The rapidly increasing computational demands for artificial intelligence (AI) have spurred the exploration of computing principles beyond conventional digital computers. Physical neural networks (PNNs) offer efficient neuromorphic information processing by harnessing the innate computational power of physical processes; however, training their weight parameters is computationally expensive. We propose a training approach for substantially reducing this training cost. Our training approach merges an optimal control method for continuous-time dynamical systems with a biologically plausible training method--direct feedback alignment. In addition to the reduction of training time, this approach achieves robust processing even under measurement errors and noise without requiring detailed system information. The effectiveness was numerically and experimentally verified in an optoelectronic delay system. Our approach significantly extends the range of physical systems practically usable as PNNs.

Related papers

• On Hardening DNNs against Noisy Computations [5.975221928631025]
  This work investigates the effectiveness of training neural networks with quantization to increase the robustness against noise. We compare these methods with noisy training, which incorporates a noise injection during training that mimics the noise encountered during inference.
  arXiv  Detail & Related papers  (2025-01-24T14:37:24Z)
• Optical training of large-scale Transformers and deep neural networks with direct feedback alignment [48.90869997343841]
  We experimentally implement a versatile and scalable training algorithm, called direct feedback alignment, on a hybrid electronic-photonic platform. An optical processing unit performs large-scale random matrix multiplications, which is the central operation of this algorithm, at speeds up to 1500 TeraOps. We study the compute scaling of our hybrid optical approach, and demonstrate a potential advantage for ultra-deep and wide neural networks.
  arXiv  Detail & Related papers  (2024-09-01T12:48:47Z)
• Neuromorphic analog circuits for robust on-chip always-on learning in spiking neural networks [1.9809266426888898]
  Mixed-signal neuromorphic systems represent a promising solution for solving extreme-edge computing tasks. Their spiking neural network circuits are optimized for processing sensory data on-line in continuous-time. We design on-chip learning circuits with short-term analog dynamics and long-term tristate discretization mechanisms.
  arXiv  Detail & Related papers  (2023-07-12T11:14:25Z)
• Backpropagation-free Training of Deep Physical Neural Networks [0.0]
  We propose a simple deep neural network architecture augmented by a biologically plausible learning algorithm, referred to as "model-free forward-forward training" We show that our method outperforms state-of-the-art hardware-aware training methods by improving training speed, decreasing digital computations, and reducing power consumption in physical systems.
  arXiv  Detail & Related papers  (2023-04-20T14:02:49Z)
• AttNS: Attention-Inspired Numerical Solving For Limited Data Scenarios [51.94807626839365]
  We propose the attention-inspired numerical solver (AttNS) to solve differential equations due to limited data.<n>AttNS is inspired by the effectiveness of attention modules in Residual Neural Networks (ResNet) in enhancing model generalization and robustness.
  arXiv  Detail & Related papers  (2023-02-05T01:39:21Z)
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks [56.35403810762512]
  Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. We study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method.
  arXiv  Detail & Related papers  (2023-02-01T04:22:59Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Physical Deep Learning with Biologically Plausible Training Method [2.5608506499175094]
  We present physical deep learning by extending a biologically plausible training algorithm called direct feedback alignment. We can emulate and accelerate the computation for this training on a simple and scalable physical system. Our results provide practical solutions for the training and acceleration of neuromorphic computation.
  arXiv  Detail & Related papers  (2022-04-01T05:46:16Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• Ps and Qs: Quantization-aware pruning for efficient low latency neural network inference [56.24109486973292]
  We study the interplay between pruning and quantization during the training of neural networks for ultra low latency applications. We find that quantization-aware pruning yields more computationally efficient models than either pruning or quantization alone for our task.
  arXiv  Detail & Related papers  (2021-02-22T19:00:05Z)
• Physical deep learning based on optimal control of dynamical systems [0.0]
  In this study, we perform pattern recognition based on the optimal control of continuous-time dynamical systems. As a key example, we apply the dynamics-based recognition approach to an optoelectronic delay system. This is in contrast to conventional multilayer neural networks, which require a large number of weight parameters to be trained.
  arXiv  Detail & Related papers  (2020-12-16T06:38:01Z)
• Spiking Neural Networks Hardware Implementations and Challenges: a Survey [53.429871539789445]
  Spiking Neural Networks are cognitive algorithms mimicking neuron and synapse operational principles. We present the state of the art of hardware implementations of spiking neural networks. We discuss the strategies employed to leverage the characteristics of these event-driven algorithms at the hardware level.
  arXiv  Detail & Related papers  (2020-05-04T13:24:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.