Deep Residual Learning in Spiking Neural Networks

• URL: http://arxiv.org/abs/2102.04159v6
• Date: Sat, 22 Jan 2022 03:14:21 GMT
• Title: Deep Residual Learning in Spiking Neural Networks
• Authors: Wei Fang, Zhaofei Yu, Yanqi Chen, Tiejun Huang, Timoth\'ee Masquelier, Yonghong Tian
• Abstract summary: Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches. Considering the huge success of ResNet in deep learning, it would be natural to train deep SNNs with residual learning. We propose spike-element-wise (SEW) ResNet to realize residual learning in deep SNNs.
• Score: 36.16846259899793
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Spiking Neural Networks (SNNs) present optimization difficulties for gradient-based approaches due to discrete binary activation and complex spatial-temporal dynamics. Considering the huge success of ResNet in deep learning, it would be natural to train deep SNNs with residual learning. Previous Spiking ResNet mimics the standard residual block in ANNs and simply replaces ReLU activation layers with spiking neurons, which suffers the degradation problem and can hardly implement residual learning. In this paper, we propose the spike-element-wise (SEW) ResNet to realize residual learning in deep SNNs. We prove that the SEW ResNet can easily implement identity mapping and overcome the vanishing/exploding gradient problems of Spiking ResNet. We evaluate our SEW ResNet on ImageNet, DVS Gesture, and CIFAR10-DVS datasets, and show that SEW ResNet outperforms the state-of-the-art directly trained SNNs in both accuracy and time-steps. Moreover, SEW ResNet can achieve higher performance by simply adding more layers, providing a simple method to train deep SNNs. To our best knowledge, this is the first time that directly training deep SNNs with more than 100 layers becomes possible. Our codes are available at https://github.com/fangwei123456/Spike-Element-Wise-ResNet.

Related papers

• An Exact Mapping From ReLU Networks to Spiking Neural Networks [3.1701886344065255]
  We propose an exact mapping from a network with Rectified Linear Units (ReLUs) to an SNN that fires exactly one spike per neuron. More generally our work shows that an arbitrary deep ReLU network can be replaced by an energy-efficient single-spike neural network without any loss of performance.
  arXiv  Detail & Related papers  (2022-12-23T18:31:09Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Deep Learning without Shortcuts: Shaping the Kernel with Tailored Rectifiers [83.74380713308605]
  We develop a new type of transformation that is fully compatible with a variant of ReLUs -- Leaky ReLUs. We show in experiments that our method, which introduces negligible extra computational cost, validation accuracies with deep vanilla networks that are competitive with ResNets.
  arXiv  Detail & Related papers  (2022-03-15T17:49:08Z)
• Advancing Residual Learning towards Powerful Deep Spiking Neural Networks [16.559670769601038]
  Residual learning and shortcuts have been evidenced as an important approach for training deep neural networks. MS-ResNet is able to significantly extend the depth of directly trained SNNs. MS-ResNet 104 achieves 76.02% accuracy on ImageNet, the first time in the domain of directly trained SNNs.
  arXiv  Detail & Related papers  (2021-12-15T05:47:21Z)
• Pruning of Deep Spiking Neural Networks through Gradient Rewiring [41.64961999525415]
  Spiking Neural Networks (SNNs) have been attached great importance due to their biological plausibility and high energy-efficiency on neuromorphic chips. Most existing methods directly apply pruning approaches in artificial neural networks (ANNs) to SNNs, which ignore the difference between ANNs and SNNs. We propose gradient rewiring (Grad R), a joint learning algorithm of connectivity and weight for SNNs, that enables us to seamlessly optimize network structure without retrain.
  arXiv  Detail & Related papers  (2021-05-11T10:05:53Z)
• S2-BNN: Bridging the Gap Between Self-Supervised Real and 1-bit Neural Networks via Guided Distribution Calibration [74.5509794733707]
  We present a novel guided learning paradigm from real-valued to distill binary networks on the final prediction distribution. Our proposed method can boost the simple contrastive learning baseline by an absolute gain of 5.515% on BNNs. Our method achieves substantial improvement over the simple contrastive learning baseline, and is even comparable to many mainstream supervised BNN methods.
  arXiv  Detail & Related papers  (2021-02-17T18:59:28Z)
• Going Deeper With Directly-Trained Larger Spiking Neural Networks [20.40894876501739]
  Spiking neural networks (SNNs) are promising in coding for bio-usible information and event-driven signal processing. However, the unique working mode of SNNs makes them more difficult to train than traditional networks. We propose a CIF-dependent batch normalization (tpladBN) method based on the emerging-temporal backproation threshold.
  arXiv  Detail & Related papers  (2020-10-29T07:15:52Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Training Deep Spiking Neural Networks [0.0]
  Brain-inspired spiking neural networks (SNNs) with neuromorphic hardware may offer orders of magnitude higher energy efficiency. We show that is is possible to train SNN with ResNet50 architecture on CIFAR100 and Imagenette object recognition datasets. The trained SNN falls behind in accuracy compared to analogous ANN but requires several orders of magnitude less inference time steps.
  arXiv  Detail & Related papers  (2020-06-08T09:47:05Z)
• Skip Connections Matter: On the Transferability of Adversarial Examples Generated with ResNets [83.12737997548645]
  Skip connections are an essential component of current state-of-the-art deep neural networks (DNNs) Use of skip connections allows easier generation of highly transferable adversarial examples. We conduct comprehensive transfer attacks against state-of-the-art DNNs including ResNets, DenseNets, Inceptions, Inception-ResNet, Squeeze-and-Excitation Network (SENet)
  arXiv  Detail & Related papers  (2020-02-14T12:09:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.