EXODUS: Stable and Efficient Training of Spiking Neural Networks

• URL: http://arxiv.org/abs/2205.10242v1
• Date: Fri, 20 May 2022 15:13:58 GMT
• Title: EXODUS: Stable and Efficient Training of Spiking Neural Networks
• Authors: Felix Christian Bauer (1), Gregor Lenz (1), Saeid Haghighatshoar (1), Sadique Sheik (1) ((1) SynSense)
• Abstract summary: Spiking Neural Networks (SNNs) are gaining significant traction in machine learning tasks where energy-efficiency is of utmost importance. Previous work by Shrestha and Orchard [ 2018] employs an efficient GPU-accelerated back-propagation algorithm called SLAYER, which speeds up training considerably. We modify SLAYER and design an algorithm called EXODUS, that accounts for the neuron reset mechanism and applies the Implicit Function Theorem (IFT) to calculate the correct gradients.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Spiking Neural Networks (SNNs) are gaining significant traction in machine learning tasks where energy-efficiency is of utmost importance. Training such networks using the state-of-the-art back-propagation through time (BPTT) is, however, very time-consuming. Previous work by Shrestha and Orchard [2018] employs an efficient GPU-accelerated back-propagation algorithm called SLAYER, which speeds up training considerably. SLAYER, however, does not take into account the neuron reset mechanism while computing the gradients, which we argue to be the source of numerical instability. To counteract this, SLAYER introduces a gradient scale hyperparameter across layers, which needs manual tuning. In this paper, (i) we modify SLAYER and design an algorithm called EXODUS, that accounts for the neuron reset mechanism and applies the Implicit Function Theorem (IFT) to calculate the correct gradients (equivalent to those computed by BPTT), (ii) we eliminate the need for ad-hoc scaling of gradients, thus, reducing the training complexity tremendously, (iii) we demonstrate, via computer simulations, that EXODUS is numerically stable and achieves a comparable or better performance than SLAYER especially in various tasks with SNNs that rely on temporal features. Our code is available at https://github.com/synsense/sinabs-exodus.

Related papers

• Efficient Event-based Delay Learning in Spiking Neural Networks [0.1350479308585481]
  Spiking Neural Networks (SNNs) are attracting increased attention as an energy-efficient alternative to traditional Neural Networks. We propose a novel event-based training method for SNNs, grounded in the EventPropProp formalism. We show that our approach uses less than half the memory of the current state-of-the-art delay-learning method and is up to 26x faster.
  arXiv  Detail & Related papers  (2025-01-13T13:44:34Z)
• DelGrad: Exact event-based gradients in spiking networks for training delays and weights [1.5226147562426895]
  Spiking neural networks (SNNs) inherently rely on the timing of signals for representing and processing information. We propose DelGrad, an event-based method to compute exact loss gradients for both synaptic weights and delays. We experimentally demonstrate the memory efficiency and accuracy benefits of adding delays to SNNs on noisy mixed-signal hardware.
  arXiv  Detail & Related papers  (2024-04-30T00:02:34Z)
• Accelerating SNN Training with Stochastic Parallelizable Spiking Neurons [1.7056768055368383]
  Spiking neural networks (SNN) are able to learn features while using less energy, especially on neuromorphic hardware. Most widely used neuron in deep learning is the temporal and Fire (LIF) neuron.
  arXiv  Detail & Related papers  (2023-06-22T04:25:27Z)
• Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks [70.75043144299168]
  Spiking Neural Networks (SNNs) are promising energy-efficient models for neuromorphic computing. We propose the Spatial Learning Through Time (SLTT) method that can achieve high performance while greatly improving training efficiency. Our method achieves state-of-the-art accuracy on ImageNet, while the memory cost and training time are reduced by more than 70% and 50%, respectively, compared with BPTT.
  arXiv  Detail & Related papers  (2023-02-28T05:01:01Z)
• Loss shaping enhances exact gradient learning with Eventprop in spiking neural networks [0.1350479308585481]
  Eventprop is an algorithm for gradient descent on exact gradients in spiking neural networks. We implement Eventprop in the GPU-enhanced Neural Networks framework. We train spiking neural networks on Spiking Heidelberg Digits and Spiking Speech Commands datasets.
  arXiv  Detail & Related papers  (2022-12-02T15:20:58Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Online Training Through Time for Spiking Neural Networks [66.7744060103562]
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. We propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning.
  arXiv  Detail & Related papers  (2022-10-09T07:47:56Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.