Gated Attention Coding for Training High-performance and Efficient Spiking Neural Networks

• URL: http://arxiv.org/abs/2308.06582v2
• Date: Tue, 4 Jun 2024 07:27:50 GMT
• Title: Gated Attention Coding for Training High-performance and Efficient Spiking Neural Networks
• Authors: Xuerui Qiu, Rui-Jie Zhu, Yuhong Chou, Zhaorui Wang, Liang-jian Deng, Guoqi Li,
• Abstract summary: Gated Attention Coding (GAC) is a plug-and-play module that leverages the multi-dimensional attention unit to efficiently encode inputs into powerful representations. GAC functions as a preprocessing layer that does not disrupt the spike-driven nature of the SNN. Experiments on CIFAR10/100 and ImageNet datasets demonstrate that GAC achieves state-of-the-art accuracy with remarkable efficiency.
• Score: 22.66931446718083
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking neural networks (SNNs) are emerging as an energy-efficient alternative to traditional artificial neural networks (ANNs) due to their unique spike-based event-driven nature. Coding is crucial in SNNs as it converts external input stimuli into spatio-temporal feature sequences. However, most existing deep SNNs rely on direct coding that generates powerless spike representation and lacks the temporal dynamics inherent in human vision. Hence, we introduce Gated Attention Coding (GAC), a plug-and-play module that leverages the multi-dimensional gated attention unit to efficiently encode inputs into powerful representations before feeding them into the SNN architecture. GAC functions as a preprocessing layer that does not disrupt the spike-driven nature of the SNN, making it amenable to efficient neuromorphic hardware implementation with minimal modifications. Through an observer model theoretical analysis, we demonstrate GAC's attention mechanism improves temporal dynamics and coding efficiency. Experiments on CIFAR10/100 and ImageNet datasets demonstrate that GAC achieves state-of-the-art accuracy with remarkable efficiency. Notably, we improve top-1 accuracy by 3.10\% on CIFAR100 with only 6-time steps and 1.07\% on ImageNet while reducing energy usage to 66.9\% of the previous works. To our best knowledge, it is the first time to explore the attention-based dynamic coding scheme in deep SNNs, with exceptional effectiveness and efficiency on large-scale datasets.The Code is available at https://github.com/bollossom/GAC.

Related papers

• Adaptive Spiking Neural Networks with Hybrid Coding [0.0]
  Spi-temporal Neural Network (SNN) is a more energy-efficient and effective neural network compared to Artificial Neural Networks (ANNs) Traditional SNNs utilize same neurons when processing input data across different time steps, limiting their ability to integrate and utilizetemporal information effectively. This paper introduces a hybrid encoding approach that not only reduces the required time steps for training but also continues to improve the overall network performance.
  arXiv  Detail & Related papers  (2024-08-22T13:58:35Z)
• TCJA-SNN: Temporal-Channel Joint Attention for Spiking Neural Networks [22.965024490694525]
  Spiking Neural Networks (SNNs) are attracting widespread interest due to their biological plausibility, energy efficiency and powerful-temporal information representation ability. We present a Temporal-Channel Joint Attention mechanism for SNNs, referred to as TCJA-SNN. The proposed TCJA-SNN framework can effectively assess the significance of spike sequence from both spatial and temporal dimensions.
  arXiv  Detail & Related papers  (2022-06-21T08:16:08Z)
• A Resource-efficient Spiking Neural Network Accelerator Supporting Emerging Neural Encoding [6.047137174639418]
  Spiking neural networks (SNNs) recently gained momentum due to their low-power multiplication-free computing. SNNs require very long spike trains (up to 1000) to reach an accuracy similar to their artificial neural network (ANN) counterparts for large models. We present a novel hardware architecture that can efficiently support SNN with emerging neural encoding.
  arXiv  Detail & Related papers  (2022-06-06T10:56:25Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Hybrid SNN-ANN: Energy-Efficient Classification and Object Detection for Event-Based Vision [64.71260357476602]
  Event-based vision sensors encode local pixel-wise brightness changes in streams of events rather than image frames. Recent progress in object recognition from event-based sensors has come from conversions of deep neural networks. We propose a hybrid architecture for end-to-end training of deep neural networks for event-based pattern recognition and object detection.
  arXiv  Detail & Related papers  (2021-12-06T23:45:58Z)
• Training Energy-Efficient Deep Spiking Neural Networks with Single-Spike Hybrid Input Encoding [5.725845886457027]
  Spiking Neural Networks (SNNs) provide higher computational efficiency in event driven neuromorphic hardware. SNNs suffer from high inference latency, resulting from inefficient input encoding and training techniques. This paper presents a training framework for low-latency energy-efficient SNNs.
  arXiv  Detail & Related papers  (2021-07-26T06:16:40Z)
• Binary Graph Neural Networks [69.51765073772226]
  Graph Neural Networks (GNNs) have emerged as a powerful and flexible framework for representation learning on irregular data. In this paper, we present and evaluate different strategies for the binarization of graph neural networks. We show that through careful design of the models, and control of the training process, binary graph neural networks can be trained at only a moderate cost in accuracy on challenging benchmarks.
  arXiv  Detail & Related papers  (2020-12-31T18:48:58Z)
• Going Deeper With Directly-Trained Larger Spiking Neural Networks [20.40894876501739]
  Spiking neural networks (SNNs) are promising in coding for bio-usible information and event-driven signal processing. However, the unique working mode of SNNs makes them more difficult to train than traditional networks. We propose a CIF-dependent batch normalization (tpladBN) method based on the emerging-temporal backproation threshold.
  arXiv  Detail & Related papers  (2020-10-29T07:15:52Z)
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning [56.83172249278467]
  We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces. We train and validate our approach directly on the Intel NNP-I chip for inference. We additionally achieve 28-78% speed-up compared to the native NNP-I compiler on all three workloads.
  arXiv  Detail & Related papers  (2020-07-14T18:50:12Z)
• Progressive Tandem Learning for Pattern Recognition with Deep Spiking Neural Networks [80.15411508088522]
  Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency. We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
  arXiv  Detail & Related papers  (2020-07-02T15:38:44Z)
• You Only Spike Once: Improving Energy-Efficient Neuromorphic Inference to ANN-Level Accuracy [51.861168222799186]
  Spiking Neural Networks (SNNs) are a type of neuromorphic, or brain-inspired network. SNNs are sparse, accessing very few weights, and typically only use addition operations instead of the more power-intensive multiply-and-accumulate operations. In this work, we aim to overcome the limitations of TTFS-encoded neuromorphic systems.
  arXiv  Detail & Related papers  (2020-06-03T15:55:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.