Optimising Event-Driven Spiking Neural Network with Regularisation and Cutoff

• URL: http://arxiv.org/abs/2301.09522v3
• Date: Tue, 5 Mar 2024 11:57:27 GMT
• Title: Optimising Event-Driven Spiking Neural Network with Regularisation and Cutoff
• Authors: Dengyu Wu and Gaojie Jin and Han Yu and Xinping Yi and Xiaowei Huang
• Abstract summary: Spiking neural network (SNN) offers promising improvements in computational efficiency. Current SNN training methodologies predominantly employ a fixed timestep approach. We propose to consider cutoff in SNN, which can terminate SNN anytime during the inference to achieve efficient inference.
• Score: 33.91830001268308
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spiking neural network (SNN), next generation of artificial neural network (ANN) that more closely mimic natural neural networks offers promising improvements in computational efficiency. However, current SNN training methodologies predominantly employ a fixed timestep approach, overlooking the potential of dynamic inference in SNN. In this paper, we strengthen the marriage between SNN and event-driven processing with a proposal to consider cutoff in SNN, which can terminate SNN anytime during the inference to achieve efficient inference. Two novel optimisation techniques are presented to achieve inference efficient SNN: a Top-K cutoff and a regularisation. The Top-K cutoff technique optimises the inference of SNN, and the regularisation are proposed to affect the training and construct SNN with optimised performance for cutoff. We conduct an extensive set of experiments on multiple benchmark frame-based datsets, such as Cifar10/100, Tiny-ImageNet and event-based datasets, including CIFAR10-DVS, N-Caltech101 and DVS128 Gesture. The experimental results demonstrate the effectiveness of our techniques in both ANN-to-SNN conversion and direct training, affirming their compatibility and potential benefits in enhancing accuracy and reducing inference timestep when integrated with existing methods. Code available: https://github.com/Dengyu-Wu/SNN-Regularisation-Cutoff

Related papers

• Spikeformer: A Novel Architecture for Training High-Performance Low-Latency Spiking Neural Network [6.8125324121155275]
  We propose a novel Transformer-based SNN,termed "Spikeformer",which outperforms its ANN counterpart on both static dataset and neuromorphic dataset. Remarkably,our Spikeformer outperforms other SNNs on ImageNet by a large margin (i.e.more than 5%) and even outperforms its ANN counterpart by 3.1% and 2.2% on DVS-Gesture and ImageNet.
  arXiv  Detail & Related papers  (2022-11-19T12:49:22Z)
• SNN2ANN: A Fast and Memory-Efficient Training Framework for Spiking Neural Networks [117.56823277328803]
  Spiking neural networks are efficient computation models for low-power environments. We propose a SNN-to-ANN (SNN2ANN) framework to train the SNN in a fast and memory-efficient way. Experiment results show that our SNN2ANN-based models perform well on the benchmark datasets.
  arXiv  Detail & Related papers  (2022-06-19T16:52:56Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Comparative Analysis of Interval Reachability for Robust Implicit and Feedforward Neural Networks [64.23331120621118]
  We use interval reachability analysis to obtain robustness guarantees for implicit neural networks (INNs) INNs are a class of implicit learning models that use implicit equations as layers. We show that our approach performs at least as well as, and generally better than, applying state-of-the-art interval bound propagation methods to INNs.
  arXiv  Detail & Related papers  (2022-04-01T03:31:27Z)
• Rethinking Pretraining as a Bridge from ANNs to SNNs [13.984523794353477]
  Spiking neural networks (SNNs) are known as a typical kind of brain-inspired models with their unique features. How to obtain a high-accuracy model has always been the main challenge in the field of SNN.
  arXiv  Detail & Related papers  (2022-03-02T14:59:57Z)
• Spatial-Temporal-Fusion BNN: Variational Bayesian Feature Layer [77.78479877473899]
  We design a spatial-temporal-fusion BNN for efficiently scaling BNNs to large models. Compared to vanilla BNNs, our approach can greatly reduce the training time and the number of parameters, which contributes to scale BNNs efficiently.
  arXiv  Detail & Related papers  (2021-12-12T17:13:14Z)
• Beyond Classification: Directly Training Spiking Neural Networks for Semantic Segmentation [5.800785186389827]
  Spiking Neural Networks (SNNs) have emerged as the low-power alternative to Artificial Neural Networks (ANNs) In this paper, we explore the SNN applications beyond classification and present semantic segmentation networks configured with spiking neurons.
  arXiv  Detail & Related papers  (2021-10-14T21:53:03Z)
• Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks [0.0]
  Spiking neural networks (SNNs) are biology-inspired artificial neural networks (ANNs) We propose a novel strategic pipeline that transfers the weights to the target SNN by combining threshold balance and soft-reset mechanisms. Our method is promising to get implanted onto embedded platforms with better support of SNNs with limited energy and memory.
  arXiv  Detail & Related papers  (2021-02-28T12:04:22Z)
• Deep Time Delay Neural Network for Speech Enhancement with Full Data Learning [60.20150317299749]
  This paper proposes a deep time delay neural network (TDNN) for speech enhancement with full data learning. To make full use of the training data, we propose a full data learning method for speech enhancement.
  arXiv  Detail & Related papers  (2020-11-11T06:32:37Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike Timing Dependent Backpropagation [10.972663738092063]
  Spiking Neural Networks (SNNs) operate with asynchronous discrete events (or spikes) We present a computationally-efficient training technique for deep SNNs. We achieve top-1 accuracy of 65.19% for ImageNet dataset on SNN with 250 time steps, which is 10X faster compared to converted SNNs with similar accuracy.
  arXiv  Detail & Related papers  (2020-05-04T19:30:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.