Memory-Efficient Reversible Spiking Neural Networks

• URL: http://arxiv.org/abs/2312.07922v1
• Date: Wed, 13 Dec 2023 06:39:49 GMT
• Title: Memory-Efficient Reversible Spiking Neural Networks
• Authors: Hong Zhang, Yu Zhang
• Abstract summary: Spiking neural networks (SNNs) are potential competitors to artificial neural networks (ANNs) SNNs require much more memory than ANNs, which impedes the training of deeper SNN models. We propose the reversible spiking neural network to reduce the memory cost of intermediate activations and membrane potentials during training.
• Score: 8.05761813203348
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking neural networks (SNNs) are potential competitors to artificial neural networks (ANNs) due to their high energy-efficiency on neuromorphic hardware. However, SNNs are unfolded over simulation time steps during the training process. Thus, SNNs require much more memory than ANNs, which impedes the training of deeper SNN models. In this paper, we propose the reversible spiking neural network to reduce the memory cost of intermediate activations and membrane potentials during training. Firstly, we extend the reversible architecture along temporal dimension and propose the reversible spiking block, which can reconstruct the computational graph and recompute all intermediate variables in forward pass with a reverse process. On this basis, we adopt the state-of-the-art SNN models to the reversible variants, namely reversible spiking ResNet (RevSResNet) and reversible spiking transformer (RevSFormer). Through experiments on static and neuromorphic datasets, we demonstrate that the memory cost per image of our reversible SNNs does not increase with the network depth. On CIFAR10 and CIFAR100 datasets, our RevSResNet37 and RevSFormer-4-384 achieve comparable accuracies and consume 3.79x and 3.00x lower GPU memory per image than their counterparts with roughly identical model complexity and parameters. We believe that this work can unleash the memory constraints in SNN training and pave the way for training extremely large and deep SNNs. The code is available at https://github.com/mi804/RevSNN.git.

Related papers

• High-Performance Temporal Reversible Spiking Neural Networks with $O(L)$ Training Memory and $O(1)$ Inference Cost [32.44622524827913]
  Spiking Neural Networks (SNNs) boost memory requirements during training and increase inference energy cost. This work proposes a novel Temporal Reversible architecture for SNNs (T-RevSNN) to jointly address the training and inference challenges. T-RevSNN achieves excellent accuracy on ImageNet, while the memory efficiency, training time acceleration, and inference energy efficiency can be significantly improved.
  arXiv  Detail & Related papers  (2024-05-26T07:26:56Z)
• Towards Zero Memory Footprint Spiking Neural Network Training [7.4331790419913455]
  Spiking Neural Networks (SNNs) process information using discrete-time events known as spikes rather than continuous values. In this paper, we introduce an innovative framework characterized by a remarkably low memory footprint. Our design is able to achieve a $mathbf58.65times$ reduction in memory usage compared to the current SNN node.
  arXiv  Detail & Related papers  (2023-08-16T19:49:24Z)
• SPIDE: A Purely Spike-based Method for Training Feedback Spiking Neural Networks [56.35403810762512]
  Spiking neural networks (SNNs) with event-based computation are promising brain-inspired models for energy-efficient applications on neuromorphic hardware. We study spike-based implicit differentiation on the equilibrium state (SPIDE) that extends the recently proposed training method.
  arXiv  Detail & Related papers  (2023-02-01T04:22:59Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Sub-bit Neural Networks: Learning to Compress and Accelerate Binary Neural Networks [72.81092567651395]
  Sub-bit Neural Networks (SNNs) are a new type of binary quantization design tailored to compress and accelerate BNNs. SNNs are trained with a kernel-aware optimization framework, which exploits binary quantization in the fine-grained convolutional kernel space. Experiments on visual recognition benchmarks and the hardware deployment on FPGA validate the great potentials of SNNs.
  arXiv  Detail & Related papers  (2021-10-18T11:30:29Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Spiking Neural Networks with Improved Inherent Recurrence Dynamics for Sequential Learning [6.417011237981518]
  Spiking neural networks (SNNs) with leaky integrate and fire (LIF) neurons can be operated in an event-driven manner. We show that SNNs can be trained for sequential tasks and propose modifications to a network of LIF neurons. We then develop a training scheme to train the proposed SNNs with improved inherent recurrence dynamics.
  arXiv  Detail & Related papers  (2021-09-04T17:13:28Z)
• Optimal Conversion of Conventional Artificial Neural Networks to Spiking Neural Networks [0.0]
  Spiking neural networks (SNNs) are biology-inspired artificial neural networks (ANNs) We propose a novel strategic pipeline that transfers the weights to the target SNN by combining threshold balance and soft-reset mechanisms. Our method is promising to get implanted onto embedded platforms with better support of SNNs with limited energy and memory.
  arXiv  Detail & Related papers  (2021-02-28T12:04:22Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• Event-Based Angular Velocity Regression with Spiking Networks [51.145071093099396]
  Spiking Neural Networks (SNNs) process information conveyed as temporal spikes rather than numeric values. We propose, for the first time, a temporal regression problem of numerical values given events from an event camera. We show that we can successfully train an SNN to perform angular velocity regression.
  arXiv  Detail & Related papers  (2020-03-05T17:37:16Z)
• RMP-SNN: Residual Membrane Potential Neuron for Enabling Deeper High-Accuracy and Low-Latency Spiking Neural Network [11.447730771403464]
  Spiking Neural Networks (SNNs) have attracted significant research interest as the third generation of artificial neural networks. We show near loss-less ANN-SNN conversion using RMP neurons for VGG-16, ResNet-20, and ResNet-34 SNNs on challenging datasets.
  arXiv  Detail & Related papers  (2020-02-25T18:19:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.