Going Deeper With Directly-Trained Larger Spiking Neural Networks

• URL: http://arxiv.org/abs/2011.05280v2
• Date: Fri, 18 Dec 2020 06:50:45 GMT
• Title: Going Deeper With Directly-Trained Larger Spiking Neural Networks
• Authors: Hanle Zheng, Yujie Wu, Lei Deng, Yifan Hu and Guoqi Li
• Abstract summary: Spiking neural networks (SNNs) are promising in coding for bio-usible information and event-driven signal processing. However, the unique working mode of SNNs makes them more difficult to train than traditional networks. We propose a CIF-dependent batch normalization (tpladBN) method based on the emerging-temporal backproation threshold.
• Score: 20.40894876501739
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking neural networks (SNNs) are promising in a bio-plausible coding for spatio-temporal information and event-driven signal processing, which is very suited for energy-efficient implementation in neuromorphic hardware. However, the unique working mode of SNNs makes them more difficult to train than traditional networks. Currently, there are two main routes to explore the training of deep SNNs with high performance. The first is to convert a pre-trained ANN model to its SNN version, which usually requires a long coding window for convergence and cannot exploit the spatio-temporal features during training for solving temporal tasks. The other is to directly train SNNs in the spatio-temporal domain. But due to the binary spike activity of the firing function and the problem of gradient vanishing or explosion, current methods are restricted to shallow architectures and thereby difficult in harnessing large-scale datasets (e.g. ImageNet). To this end, we propose a threshold-dependent batch normalization (tdBN) method based on the emerging spatio-temporal backpropagation, termed "STBP-tdBN", enabling direct training of a very deep SNN and the efficient implementation of its inference on neuromorphic hardware. With the proposed method and elaborated shortcut connection, we significantly extend directly-trained SNNs from a shallow structure ( < 10 layer) to a very deep structure (50 layers). Furthermore, we theoretically analyze the effectiveness of our method based on "Block Dynamical Isometry" theory. Finally, we report superior accuracy results including 93.15 % on CIFAR-10, 67.8 % on DVS-CIFAR10, and 67.05% on ImageNet with very few timesteps. To our best knowledge, it's the first time to explore the directly-trained deep SNNs with high performance on ImageNet.

Related papers

• Deep Directly-Trained Spiking Neural Networks for Object Detection [20.594942840081757]
  EMS-YOLO is a novel directly-trained SNN framework for object detection. We design a full-spike residual block, EMS-ResNet, which can effectively extend the depth of the directly-trained SNN with low power consumption. It is shown that our model could achieve comparable performance to the ANN with the same architecture while consuming 5.83 times less energy.
  arXiv  Detail & Related papers  (2023-07-21T08:10:26Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Online Training Through Time for Spiking Neural Networks [66.7744060103562]
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. We propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning.
  arXiv  Detail & Related papers  (2022-10-09T07:47:56Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Rethinking Pretraining as a Bridge from ANNs to SNNs [13.984523794353477]
  Spiking neural networks (SNNs) are known as a typical kind of brain-inspired models with their unique features. How to obtain a high-accuracy model has always been the main challenge in the field of SNN.
  arXiv  Detail & Related papers  (2022-03-02T14:59:57Z)
• Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layer [77.78479877473899]
  We design a spatial-temporal-fusion BNN for efficiently scaling BNNs to large models. Compared to vanilla BNNs, our approach can greatly reduce the training time and the number of parameters, which contributes to scale BNNs efficiently.
  arXiv  Detail & Related papers  (2021-12-12T17:13:14Z)
• Sparse Spiking Gradient Descent [2.741266294612776]
  We present the first sparse SNN backpropagation algorithm which achieves the same or better accuracy as current state of the art methods. We show the effectiveness of our method on real datasets of varying complexity.
  arXiv  Detail & Related papers  (2021-05-18T20:00:55Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation [10.972663738092063]
  Spiking Neural Networks (SNNs) operate with asynchronous discrete events (or spikes) We present a computationally-efficient training technique for deep SNNs. We achieve top-1 accuracy of 65.19% for ImageNet dataset on SNN with 250 time steps, which is 10X faster compared to converted SNNs with similar accuracy.
  arXiv  Detail & Related papers  (2020-05-04T19:30:43Z)
• T2FSNN: Deep Spiking Neural Networks with Time-to-first-spike Coding [26.654533157221973]
  This paper introduces the concept of time-to-first-spike coding into deep SNNs using the kernel-based dynamic threshold and dendrite to overcome the drawback. According to our results, the proposed methods can reduce inference latency and number of spikes to 22% and less than 1%, compared to those of burst coding.
  arXiv  Detail & Related papers  (2020-03-26T04:39:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.