Efficient Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment

• URL: http://arxiv.org/abs/2501.15925v1
• Date: Mon, 27 Jan 2025 10:22:38 GMT
• Title: Efficient Distillation of Deep Spiking Neural Networks for Full-Range Timestep Deployment
• Authors: Chengting Yu, Xiaochen Zhao, Lei Liu, Shu Yang, Gaoang Wang, Erping Li, Aili Wang,
• Abstract summary: Spiking Neural Networks (SNNs) are emerging as a brain-inspired alternative to traditional Artificial Neural Networks (ANNs) Despite this, SNNs often suffer from accuracy compared to ANNs and face deployment challenges due to inference timesteps, which require retraining for adjustments, limiting operational flexibility. We propose a novel distillation framework for deep SNNs that optimize performance across full-range timesteps without specific retraining, enhancing both efficacy and adaptability.
• Score: 10.026742974971189
• License:
• Abstract: Spiking Neural Networks (SNNs) are emerging as a brain-inspired alternative to traditional Artificial Neural Networks (ANNs), prized for their potential energy efficiency on neuromorphic hardware. Despite this, SNNs often suffer from accuracy degradation compared to ANNs and face deployment challenges due to fixed inference timesteps, which require retraining for adjustments, limiting operational flexibility. To address these issues, our work considers the spatio-temporal property inherent in SNNs, and proposes a novel distillation framework for deep SNNs that optimizes performance across full-range timesteps without specific retraining, enhancing both efficacy and deployment adaptability. We provide both theoretical analysis and empirical validations to illustrate that training guarantees the convergence of all implicit models across full-range timesteps. Experimental results on CIFAR-10, CIFAR-100, CIFAR10-DVS, and ImageNet demonstrate state-of-the-art performance among distillation-based SNNs training methods.

Related papers

• Temporal Misalignment and Probabilistic Neurons [17.73940693302129]
  Spiking Neural Networks (SNNs) offer a more energy-efficient alternative to Artificial Neural Networks (ANNs) In this work, we identify a phenomenon in the ANN-SNN conversion framework, termed temporal misalignment. We introduce biologically plausible two-phase probabilistic (TPP) spiking neurons, further enhancing the conversion process.
  arXiv  Detail & Related papers  (2025-02-20T12:09:30Z)
• Training a General Spiking Neural Network with Improved Efficiency and Minimum Latency [4.503744528661997]
  Spiking Neural Networks (SNNs) operate in an event-driven manner and employ binary spike representation. This paper proposes a general training framework that enhances feature learning and activation efficiency within a limited time step.
  arXiv  Detail & Related papers  (2024-01-05T09:54:44Z)
• Skip Connections in Spiking Neural Networks: An Analysis of Their Effect on Network Training [0.8602553195689513]
  Spiking neural networks (SNNs) have gained attention as a promising alternative to traditional artificial neural networks (ANNs) In this paper, we study the impact of skip connections on SNNs and propose a hyper parameter optimization technique that adapts models from ANN to SNN. We demonstrate that optimizing the position, type, and number of skip connections can significantly improve the accuracy and efficiency of SNNs.
  arXiv  Detail & Related papers  (2023-03-23T07:57:32Z)
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks [56.35403810762512]
  Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. We study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method.
  arXiv  Detail & Related papers  (2023-02-01T04:22:59Z)
• Training Spiking Neural Networks with Local Tandem Learning [96.32026780517097]
  Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efficient than their predecessors. In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL) We demonstrate rapid network convergence within five training epochs on the CIFAR-10 dataset while having low computational complexity.
  arXiv  Detail & Related papers  (2022-10-10T10:05:00Z)
• Online Training Through Time for Spiking Neural Networks [66.7744060103562]
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. We propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning.
  arXiv  Detail & Related papers  (2022-10-09T07:47:56Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Temporal Efficient Training of Spiking Neural Network via Gradient Re-weighting [29.685909045226847]
  Brain-inspired spiking neuron networks (SNNs) have attracted widespread research interest because of their event-driven and energy-efficient characteristics. Current direct training approach with surrogate gradient results in SNNs with poor generalizability. We introduce the temporal efficient training (TET) approach to compensate for the loss of momentum in the gradient descent with SG.
  arXiv  Detail & Related papers  (2022-02-24T08:02:37Z)
• HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noise [13.904091056365765]
  We present an SNN training algorithm that uses crafted input noise and incurs no additional training time. Compared to standard trained direct input SNNs, our trained models yield improved classification accuracy of up to 13.7%. Our models also outperform inherently robust SNNs trained on rate-coded inputs with improved or similar classification performance on attack-generated images.
  arXiv  Detail & Related papers  (2021-10-06T16:48:48Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.