LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks

• URL: http://arxiv.org/abs/2304.09101v1
• Date: Mon, 17 Apr 2023 03:49:35 GMT
• Title: LaSNN: Layer-wise ANN-to-SNN Distillation for Effective and Efficient Training in Deep Spiking Neural Networks
• Authors: Di Hong, Jiangrong Shen, Yu Qi, Yueming Wang
• Abstract summary: Spiking Neural Networks (SNNs) are biologically realistic and practically promising in low-power because of their event-driven mechanism. A conversion scheme is proposed to obtain competitive accuracy by mapping trained ANNs' parameters to SNNs with the same structures. A novel SNN training framework is proposed, namely layer-wise ANN-to-SNN knowledge distillation (LaSNN)
• Score: 7.0691139514420005
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking Neural Networks (SNNs) are biologically realistic and practically promising in low-power computation because of their event-driven mechanism. Usually, the training of SNNs suffers accuracy loss on various tasks, yielding an inferior performance compared with ANNs. A conversion scheme is proposed to obtain competitive accuracy by mapping trained ANNs' parameters to SNNs with the same structures. However, an enormous number of time steps are required for these converted SNNs, thus losing the energy-efficient benefit. Utilizing both the accuracy advantages of ANNs and the computing efficiency of SNNs, a novel SNN training framework is proposed, namely layer-wise ANN-to-SNN knowledge distillation (LaSNN). In order to achieve competitive accuracy and reduced inference latency, LaSNN transfers the learning from a well-trained ANN to a small SNN by distilling the knowledge other than converting the parameters of ANN. The information gap between heterogeneous ANN and SNN is bridged by introducing the attention scheme, the knowledge in an ANN is effectively compressed and then efficiently transferred by utilizing our layer-wise distillation paradigm. We conduct detailed experiments to demonstrate the effectiveness, efficacy, and scalability of LaSNN on three benchmark data sets (CIFAR-10, CIFAR-100, and Tiny ImageNet). We achieve competitive top-1 accuracy compared to ANNs and 20x faster inference than converted SNNs with similar performance. More importantly, LaSNN is dexterous and extensible that can be effortlessly developed for SNNs with different architectures/depths and input encoding methods, contributing to their potential development.

Related papers

• NAS-BNN: Neural Architecture Search for Binary Neural Networks [55.058512316210056]
  We propose a novel neural architecture search scheme for binary neural networks, named NAS-BNN. Our discovered binary model family outperforms previous BNNs for a wide range of operations (OPs) from 20M to 200M. In addition, we validate the transferability of these searched BNNs on the object detection task, and our binary detectors with the searched BNNs achieve a novel state-of-the-art result, e.g., 31.6% mAP with 370M OPs, on MS dataset.
  arXiv  Detail & Related papers  (2024-08-28T02:17:58Z)
• Enhancing Adversarial Robustness in SNNs with Sparse Gradients [46.15229142258264]
  Spiking Neural Networks (SNNs) have attracted great attention for their energy-efficient operations and biologically inspired structures. Existing techniques, whether adapted from ANNs or specifically designed for SNNs, exhibit limitations in training SNNs or defending against strong attacks. We propose a novel approach to enhance the robustness of SNNs through gradient sparsity regularization.
  arXiv  Detail & Related papers  (2024-05-30T05:39:27Z)
• Joint A-SNN: Joint Training of Artificial and Spiking Neural Networks via Self-Distillation and Weight Factorization [12.1610509770913]
  Spiking Neural Networks (SNNs) mimic the spiking nature of brain neurons. We propose a joint training framework of ANN and SNN, in which the ANN can guide the SNN's optimization. Our method consistently outperforms many other state-of-the-art training methods.
  arXiv  Detail & Related papers  (2023-05-03T13:12:17Z)
• Optimal ANN-SNN Conversion for High-accuracy and Ultra-low-latency Spiking Neural Networks [22.532709609646066]
  Spiking Neural Networks (SNNs) have gained great attraction due to their distinctive properties of low power consumption and fast inference on neuromorphic hardware. As the most effective method to get deep SNNs, ANN-SNN conversion has achieved comparable performance as ANNs on large-scale datasets. In this paper, we theoretically analyze ANN-SNN conversion error and derive the estimated activation function of SNNs. We prove that the expected conversion error between SNNs and ANNs is zero, enabling us to achieve high-accuracy and ultra-low-latency SNN
  arXiv  Detail & Related papers  (2023-03-08T03:04:53Z)
• SNN2ANN: A Fast and Memory-Efficient Training Framework for Spiking Neural Networks [117.56823277328803]
  Spiking neural networks are efficient computation models for low-power environments. We propose a SNN-to-ANN (SNN2ANN) framework to train the SNN in a fast and memory-efficient way. Experiment results show that our SNN2ANN-based models perform well on the benchmark datasets.
  arXiv  Detail & Related papers  (2022-06-19T16:52:56Z)
• Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation [5.800785186389827]
  Spiking Neural Networks (SNNs) have emerged as the low-power alternative to Artificial Neural Networks (ANNs) In this paper, we explore the SNN applications beyond classification and present semantic segmentation networks configured with spiking neurons.
  arXiv  Detail & Related papers  (2021-10-14T21:53:03Z)
• Optimal ANN-SNN Conversion for Fast and Accurate Inference in Deep Spiking Neural Networks [43.046402416604245]
  Spiking Neural Networks (SNNs) are bio-inspired energy-efficient neural networks. In this paper, we theoretically analyze ANN-SNN conversion and derive sufficient conditions of the optimal conversion. We show that the proposed method achieves near loss less conversion with VGG-16, PreActResNet-18, and deeper structures.
  arXiv  Detail & Related papers  (2021-05-25T04:15:06Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Training Deep Spiking Neural Networks [0.0]
  Brain-inspired spiking neural networks (SNNs) with neuromorphic hardware may offer orders of magnitude higher energy efficiency. We show that is is possible to train SNN with ResNet50 architecture on CIFAR100 and Imagenette object recognition datasets. The trained SNN falls behind in accuracy compared to analogous ANN but requires several orders of magnitude less inference time steps.
  arXiv  Detail & Related papers  (2020-06-08T09:47:05Z)
• You Only Spike Once: Improving Energy-Efficient Neuromorphic Inference to ANN-Level Accuracy [51.861168222799186]
  Spiking Neural Networks (SNNs) are a type of neuromorphic, or brain-inspired network. SNNs are sparse, accessing very few weights, and typically only use addition operations instead of the more power-intensive multiply-and-accumulate operations. In this work, we aim to overcome the limitations of TTFS-encoded neuromorphic systems.
  arXiv  Detail & Related papers  (2020-06-03T15:55:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.