Comprehensive Online Training and Deployment for Spiking Neural Networks

• URL: http://arxiv.org/abs/2410.07547v1
• Date: Thu, 10 Oct 2024 02:39:22 GMT
• Title: Comprehensive Online Training and Deployment for Spiking Neural Networks
• Authors: Zecheng Hao, Yifan Huang, Zijie Xu, Zhaofei Yu, Tiejun Huang,
• Abstract summary: Spiking Neural Networks (SNNs) are considered to have enormous potential in the future development of Artificial Intelligence (AI) The current proposed online training methods cannot tackle the inseparability problem of temporal dependent gradients. We propose Efficient Multi-Precision Firing (EM-PF) model, which is a family of advanced spiking models based on floating-point spikes and binary synaptic weights.
• Score: 40.255762156745405
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking Neural Networks (SNNs) are considered to have enormous potential in the future development of Artificial Intelligence (AI) due to their brain-inspired and energy-efficient properties. In the current supervised learning domain of SNNs, compared to vanilla Spatial-Temporal Back-propagation (STBP) training, online training can effectively overcome the risk of GPU memory explosion and has received widespread academic attention. However, the current proposed online training methods cannot tackle the inseparability problem of temporal dependent gradients and merely aim to optimize the training memory, resulting in no performance advantages compared to the STBP training models in the inference phase. To address the aforementioned challenges, we propose Efficient Multi-Precision Firing (EM-PF) model, which is a family of advanced spiking models based on floating-point spikes and binary synaptic weights. We point out that EM-PF model can effectively separate temporal gradients and achieve full-stage optimization towards computation speed and memory footprint. Experimental results have demonstrated that EM-PF model can be flexibly combined with various techniques including random back-propagation, parallel computation and channel attention mechanism, to achieve state-of-the-art performance with extremely low computational overhead in the field of online learning.

Related papers

• Online Pseudo-Zeroth-Order Training of Neuromorphic Spiking Neural Networks [69.2642802272367]
  Brain-inspired neuromorphic computing with spiking neural networks (SNNs) is a promising energy-efficient computational approach. Most recent methods leverage spatial and temporal backpropagation (BP), not adhering to neuromorphic properties. We propose a novel method, online pseudo-zeroth-order (OPZO) training.
  arXiv  Detail & Related papers  (2024-07-17T12:09:00Z)
• Efficient Online Learning for Networks of Two-Compartment Spiking Neurons [23.720523101102593]
  We present a novel online learning method specifically tailored for networks of TC-LIF neurons. We also propose a refined TC-LIF neuron model called Adaptive TC-LIF, which is carefully designed to enhance temporal information integration. Our approach successfully preserves the superior sequential modeling capabilities of the TC-LIF neuron while incorporating the training efficiency and hardware friendliness of online learning.
  arXiv  Detail & Related papers  (2024-02-25T03:15:12Z)
• Parallel Spiking Unit for Efficient Training of Spiking Neural Networks [8.912926151352888]
  Spiking Neural Networks (SNNs) are used to advance artificial intelligence. SNNs are hampered by their inherent sequential computational dependency. This paper introduces the innovative Parallel Spiking Unit (PSU) and its two derivatives, the Input-aware PSU (IPSU) and Reset-aware PSU (RPSU) These variants skillfully decouple the leaky integration and firing mechanisms in spiking neurons while probabilistically managing the reset process.
  arXiv  Detail & Related papers  (2024-02-01T09:36:26Z)
• Estimating Post-Synaptic Effects for Online Training of Feed-Forward SNNs [0.27016900604393124]
  Facilitating online learning in spiking neural networks (SNNs) is a key step in developing event-based models. We propose Online Training with Postsynaptic Estimates (OTPE) for training feed-forward SNNs. We show improved scaling for multi-layer networks using a novel approximation of temporal effects on the subsequent layer's activity.
  arXiv  Detail & Related papers  (2023-11-07T16:53:39Z)
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks [56.35403810762512]
  Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. We study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method.
  arXiv  Detail & Related papers  (2023-02-01T04:22:59Z)
• Dual adaptive training of photonic neural networks [30.86507809437016]
  Photonic neural network (PNN) computes with photons instead of electrons to feature low latency, high energy efficiency, and high parallelism. Existing training approaches cannot address the extensive accumulation of systematic errors in large-scale PNNs. We propose dual adaptive training ( DAT) that allows the PNN model to adapt to substantial systematic errors.
  arXiv  Detail & Related papers  (2022-12-09T05:03:45Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Online Training Through Time for Spiking Neural Networks [66.7744060103562]
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. We propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning.
  arXiv  Detail & Related papers  (2022-10-09T07:47:56Z)
• Online Training of Spiking Recurrent Neural Networks with Phase-Change Memory Synapses [1.9809266426888898]
  Training spiking neural networks (RNNs) on dedicated neuromorphic hardware is still an open challenge. We present a simulation framework of differential-architecture arrays based on an accurate and comprehensive Phase-Change Memory (PCM) device model. We train a spiking RNN whose weights are emulated in the presented simulation framework, using a recently proposed e-prop learning rule.
  arXiv  Detail & Related papers  (2021-08-04T01:24:17Z)
• Understanding the Effects of Data Parallelism and Sparsity on Neural Network Training [126.49572353148262]
  We study two factors in neural network training: data parallelism and sparsity. Despite their promising benefits, understanding of their effects on neural network training remains elusive.
  arXiv  Detail & Related papers  (2020-03-25T10:49:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.