Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks

• URL: http://arxiv.org/abs/2406.07862v1
• Date: Wed, 12 Jun 2024 04:30:40 GMT
• Title: Self-Distillation Learning Based on Temporal-Spatial Consistency for Spiking Neural Networks
• Authors: Lin Zuo, Yongqi Ding, Mengmeng Jing, Kunshan Yang, Yunqian Yu,
• Abstract summary: Spiking neural networks (SNNs) have attracted considerable attention for their event-driven, low-power characteristics and high biological interpretability. Recent research has improved the performance of the SNN model with a pre-trained teacher model. In this paper, we explore cost-effective self-distillation learning of SNNs to circumvent these concerns.
• Score: 3.7748662901422807
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Spiking neural networks (SNNs) have attracted considerable attention for their event-driven, low-power characteristics and high biological interpretability. Inspired by knowledge distillation (KD), recent research has improved the performance of the SNN model with a pre-trained teacher model. However, additional teacher models require significant computational resources, and it is tedious to manually define the appropriate teacher network architecture. In this paper, we explore cost-effective self-distillation learning of SNNs to circumvent these concerns. Without an explicit defined teacher, the SNN generates pseudo-labels and learns consistency during training. On the one hand, we extend the timestep of the SNN during training to create an implicit temporal ``teacher" that guides the learning of the original ``student", i.e., the temporal self-distillation. On the other hand, we guide the output of the weak classifier at the intermediate stage by the final output of the SNN, i.e., the spatial self-distillation. Our temporal-spatial self-distillation (TSSD) learning method does not introduce any inference overhead and has excellent generalization ability. Extensive experiments on the static image datasets CIFAR10/100 and ImageNet as well as the neuromorphic datasets CIFAR10-DVS and DVS-Gesture validate the superior performance of the TSSD method. This paper presents a novel manner of fusing SNNs with KD, providing insights into high-performance SNN learning methods.

Related papers

• Towards Low-latency Event-based Visual Recognition with Hybrid Step-wise Distillation Spiking Neural Networks [50.32980443749865]
  Spiking neural networks (SNNs) have garnered significant attention for their low power consumption and high biologicalability. Current SNNs struggle to balance accuracy and latency in neuromorphic datasets. We propose Step-wise Distillation (HSD) method, tailored for neuromorphic datasets.
  arXiv  Detail & Related papers  (2024-09-19T06:52:34Z)
• BKDSNN: Enhancing the Performance of Learning-based Spiking Neural Networks Training with Blurred Knowledge Distillation [20.34272550256856]
  Spiking neural networks (SNNs) mimic biological neural system to convey information via discrete spikes. Our work achieves state-of-the-art performance for training SNNs on both static and neuromorphic datasets.
  arXiv  Detail & Related papers  (2024-07-12T08:17:24Z)
• CLIF: Complementary Leaky Integrate-and-Fire Neuron for Spiking Neural Networks [5.587069105667678]
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. It remains a challenge to train SNNs due to their undifferentiable spiking mechanism. We propose Leaky Integrate-and-Fire Neuron-based SNNs and Complementary Leaky Integrate-and-Fire Neuron.
  arXiv  Detail & Related papers  (2024-02-07T08:51:57Z)
• Enhancing Efficient Continual Learning with Dynamic Structure Development of Spiking Neural Networks [6.407825206595442]
  Children possess the ability to learn multiple cognitive tasks sequentially. Existing continual learning frameworks are usually applicable to Deep Neural Networks (DNNs) We propose Dynamic Structure Development of Spiking Neural Networks (DSD-SNN) for efficient and adaptive continual learning.
  arXiv  Detail & Related papers  (2023-08-09T07:36:40Z)
• Temporal Contrastive Learning for Spiking Neural Networks [23.963069990569714]
  Biologically inspired neural networks (SNNs) have garnered considerable attention due to their low-energy consumption and better-temporal information processing capabilities. We propose a novel method to obtain SNNs with low latency and high performance by incorporating contrastive supervision with temporal domain information.
  arXiv  Detail & Related papers  (2023-05-23T10:31:46Z)
• Constructing Deep Spiking Neural Networks from Artificial Neural Networks with Knowledge Distillation [20.487853773309563]
  Spiking neural networks (SNNs) are well known as the brain-inspired models with high computing efficiency. We propose a novel method of constructing deep SNN models with knowledge distillation (KD)
  arXiv  Detail & Related papers  (2023-04-12T05:57:21Z)
• Training Spiking Neural Networks with Local Tandem Learning [96.32026780517097]
  Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efficient than their predecessors. In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL) We demonstrate rapid network convergence within five training epochs on the CIFAR-10 dataset while having low computational complexity.
  arXiv  Detail & Related papers  (2022-10-10T10:05:00Z)
• Online Training Through Time for Spiking Neural Networks [66.7744060103562]
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. We propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning.
  arXiv  Detail & Related papers  (2022-10-09T07:47:56Z)
• Deep Time Delay Neural Network for Speech Enhancement with Full Data Learning [60.20150317299749]
  This paper proposes a deep time delay neural network (TDNN) for speech enhancement with full data learning. To make full use of the training data, we propose a full data learning method for speech enhancement.
  arXiv  Detail & Related papers  (2020-11-11T06:32:37Z)
• On Self-Distilling Graph Neural Network [64.00508355508106]
  We propose the first teacher-free knowledge distillation method for GNNs, termed GNN Self-Distillation (GNN-SD) The method is built upon the proposed neighborhood discrepancy rate (NDR), which quantifies the non-smoothness of the embedded graph in an efficient way. We also summarize a generic GNN-SD framework that could be exploited to induce other distillation strategies.
  arXiv  Detail & Related papers  (2020-11-04T12:29:33Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.