Towards Zero Memory Footprint Spiking Neural Network Training
- URL: http://arxiv.org/abs/2308.08649v1
- Date: Wed, 16 Aug 2023 19:49:24 GMT
- Title: Towards Zero Memory Footprint Spiking Neural Network Training
- Authors: Bin Lei, Sheng Lin, Pei-Hung Lin, Chunhua Liao, Caiwen Ding
- Abstract summary: Spiking Neural Networks (SNNs) process information using discrete-time events known as spikes rather than continuous values.
In this paper, we introduce an innovative framework characterized by a remarkably low memory footprint.
Our design is able to achieve a $mathbf58.65times$ reduction in memory usage compared to the current SNN node.
- Score: 7.4331790419913455
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Biologically-inspired Spiking Neural Networks (SNNs), processing information
using discrete-time events known as spikes rather than continuous values, have
garnered significant attention due to their hardware-friendly and
energy-efficient characteristics. However, the training of SNNs necessitates a
considerably large memory footprint, given the additional storage requirements
for spikes or events, leading to a complex structure and dynamic setup. In this
paper, to address memory constraint in SNN training, we introduce an innovative
framework, characterized by a remarkably low memory footprint. We \textbf{(i)}
design a reversible SNN node that retains a high level of accuracy. Our design
is able to achieve a $\mathbf{58.65\times}$ reduction in memory usage compared
to the current SNN node. We \textbf{(ii)} propose a unique algorithm to
streamline the backpropagation process of our reversible SNN node. This
significantly trims the backward Floating Point Operations Per Second (FLOPs),
thereby accelerating the training process in comparison to current reversible
layer backpropagation method. By using our algorithm, the training time is able
to be curtailed by $\mathbf{23.8\%}$ relative to existing reversible layer
architectures.
Related papers
- High-Performance Temporal Reversible Spiking Neural Networks with $O(L)$ Training Memory and $O(1)$ Inference Cost [32.44622524827913]
Spiking Neural Networks (SNNs) boost memory requirements during training and increase inference energy cost.
This work proposes a novel Temporal Reversible architecture for SNNs (T-RevSNN) to jointly address the training and inference challenges.
T-RevSNN achieves excellent accuracy on ImageNet, while the memory efficiency, training time acceleration, and inference energy efficiency can be significantly improved.
arXiv Detail & Related papers (2024-05-26T07:26:56Z) - Topology-aware Embedding Memory for Continual Learning on Expanding Networks [63.35819388164267]
We present a framework to tackle the memory explosion problem using memory replay techniques.
PDGNNs with Topology-aware Embedding Memory (TEM) significantly outperform state-of-the-art techniques.
arXiv Detail & Related papers (2024-01-24T03:03:17Z) - TT-SNN: Tensor Train Decomposition for Efficient Spiking Neural Network
Training [27.565726483503838]
We introduce Train Decomposition for Spiking Neural Networks (TT-SNN)
TT-SNN reduces model size through trainable weight decomposition, resulting in reduced storage, FLOPs, and latency.
We also propose a parallel computation as an alternative to the typical sequential tensor computation.
arXiv Detail & Related papers (2024-01-15T23:08:19Z) - Memory-Efficient Reversible Spiking Neural Networks [8.05761813203348]
Spiking neural networks (SNNs) are potential competitors to artificial neural networks (ANNs)
SNNs require much more memory than ANNs, which impedes the training of deeper SNN models.
We propose the reversible spiking neural network to reduce the memory cost of intermediate activations and membrane potentials during training.
arXiv Detail & Related papers (2023-12-13T06:39:49Z) - Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency.
We present an IPU-optimized release of our custom SNN Python package, snnTorch.
arXiv Detail & Related papers (2022-11-19T15:44:08Z) - A temporally and spatially local spike-based backpropagation algorithm
to enable training in hardware [0.0]
Spiking Neural Networks (SNNs) have emerged as a hardware efficient architecture for classification tasks.
There have been several attempts to adopt the powerful backpropagation (BP) technique used in non-spiking artificial neural networks (ANNs)
arXiv Detail & Related papers (2022-07-20T08:57:53Z) - Sub-bit Neural Networks: Learning to Compress and Accelerate Binary
Neural Networks [72.81092567651395]
Sub-bit Neural Networks (SNNs) are a new type of binary quantization design tailored to compress and accelerate BNNs.
SNNs are trained with a kernel-aware optimization framework, which exploits binary quantization in the fine-grained convolutional kernel space.
Experiments on visual recognition benchmarks and the hardware deployment on FPGA validate the great potentials of SNNs.
arXiv Detail & Related papers (2021-10-18T11:30:29Z) - ActNN: Reducing Training Memory Footprint via 2-Bit Activation
Compressed Training [68.63354877166756]
ActNN is a memory-efficient training framework that stores randomly quantized activations for back propagation.
ActNN reduces the memory footprint of the activation by 12x, and it enables training with a 6.6x to 14x larger batch size.
arXiv Detail & Related papers (2021-04-29T05:50:54Z) - Binary Graph Neural Networks [69.51765073772226]
Graph Neural Networks (GNNs) have emerged as a powerful and flexible framework for representation learning on irregular data.
In this paper, we present and evaluate different strategies for the binarization of graph neural networks.
We show that through careful design of the models, and control of the training process, binary graph neural networks can be trained at only a moderate cost in accuracy on challenging benchmarks.
arXiv Detail & Related papers (2020-12-31T18:48:58Z) - Progressive Tandem Learning for Pattern Recognition with Deep Spiking
Neural Networks [80.15411508088522]
Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency.
We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
arXiv Detail & Related papers (2020-07-02T15:38:44Z) - Enabling Deep Spiking Neural Networks with Hybrid Conversion and Spike
Timing Dependent Backpropagation [10.972663738092063]
Spiking Neural Networks (SNNs) operate with asynchronous discrete events (or spikes)
We present a computationally-efficient training technique for deep SNNs.
We achieve top-1 accuracy of 65.19% for ImageNet dataset on SNN with 250 time steps, which is 10X faster compared to converted SNNs with similar accuracy.
arXiv Detail & Related papers (2020-05-04T19:30:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.