Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticket

• URL: http://arxiv.org/abs/2401.02020v1
• Date: Thu, 4 Jan 2024 01:33:33 GMT
• Title: Spikformer V2: Join the High Accuracy Club on ImageNet with an SNN Ticket
• Authors: Zhaokun Zhou, Kaiwei Che, Wei Fang, Keyu Tian, Yuesheng Zhu, Shuicheng Yan, Yonghong Tian, Li Yuan
• Abstract summary: Spiking Neural Networks (SNNs) face the challenge of limited performance. Self-attention mechanism, which is the cornerstone of the high-performance Transformer, is absent in existing SNNs. We propose a novel Spiking Self-Attention (SSA) and Spiking Transformer (Spikformer)
• Score: 81.9471033944819
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking Neural Networks (SNNs), known for their biologically plausible architecture, face the challenge of limited performance. The self-attention mechanism, which is the cornerstone of the high-performance Transformer and also a biologically inspired structure, is absent in existing SNNs. To this end, we explore the potential of leveraging both self-attention capability and biological properties of SNNs, and propose a novel Spiking Self-Attention (SSA) and Spiking Transformer (Spikformer). The SSA mechanism eliminates the need for softmax and captures the sparse visual feature employing spike-based Query, Key, and Value. This sparse computation without multiplication makes SSA efficient and energy-saving. Further, we develop a Spiking Convolutional Stem (SCS) with supplementary convolutional layers to enhance the architecture of Spikformer. The Spikformer enhanced with the SCS is referred to as Spikformer V2. To train larger and deeper Spikformer V2, we introduce a pioneering exploration of Self-Supervised Learning (SSL) within the SNN. Specifically, we pre-train Spikformer V2 with masking and reconstruction style inspired by the mainstream self-supervised Transformer, and then finetune the Spikformer V2 on the image classification on ImageNet. Extensive experiments show that Spikformer V2 outperforms other previous surrogate training and ANN2SNN methods. An 8-layer Spikformer V2 achieves an accuracy of 80.38% using 4 time steps, and after SSL, a 172M 16-layer Spikformer V2 reaches an accuracy of 81.10% with just 1 time step. To the best of our knowledge, this is the first time that the SNN achieves 80+% accuracy on ImageNet. The code will be available at Spikformer V2.

Related papers

• SpikingJelly: An open-source machine learning infrastructure platform for spike-based intelligence [51.6943465041708]
  Spiking neural networks (SNNs) aim to realize brain-inspired intelligence on neuromorphic chips with high energy efficiency. We contribute a full-stack toolkit for pre-processing neuromorphic datasets, building deep SNNs, optimizing their parameters, and deploying SNNs on neuromorphic chips.
  arXiv  Detail & Related papers  (2023-10-25T13:15:17Z)
• Attention-free Spikformer: Mixing Spike Sequences with Simple Linear Transforms [16.54314950692779]
  Spikformer integrates self-attention capability and the biological properties of Spiking Neural Networks (SNNs) It introduces a Spiking Self-Attention (SSA) module to mix sparse visual features using spike-form Query, Key, and Value. We conduct extensive experiments on image classification using both neuromorphic and static datasets.
  arXiv  Detail & Related papers  (2023-08-02T11:41:54Z)
• Deep Multi-Threshold Spiking-UNet for Image Processing [51.88730892920031]
  This paper introduces the novel concept of Spiking-UNet for image processing, which combines the power of Spiking Neural Networks (SNNs) with the U-Net architecture. To achieve an efficient Spiking-UNet, we face two primary challenges: ensuring high-fidelity information propagation through the network via spikes and formulating an effective training strategy. Experimental results show that, on image segmentation and denoising, our Spiking-UNet achieves comparable performance to its non-spiking counterpart.
  arXiv  Detail & Related papers  (2023-07-20T16:00:19Z)
• Auto-Spikformer: Spikformer Architecture Search [22.332981906087785]
  Self-attention mechanisms have been integrated into Spiking Neural Networks (SNNs) Recent advancements in SNN architecture, such as Spikformer, have demonstrated promising outcomes. We propose Auto-Spikformer, a one-shot Transformer Architecture Search (TAS) method, which automates the quest for an optimized Spikformer architecture.
  arXiv  Detail & Related papers  (2023-06-01T15:35:26Z)
• Spikingformer: Spike-driven Residual Learning for Transformer-based Spiking Neural Network [19.932683405796126]
  Spiking neural networks (SNNs) offer a promising energy-efficient alternative to artificial neural networks. SNNs suffer from non-spike computations caused by the structure of their residual connection. We develop Spikingformer, a pure transformer-based spiking neural network.
  arXiv  Detail & Related papers  (2023-04-24T09:44:24Z)
• Spikformer: When Spiking Neural Network Meets Transformer [102.91330530210037]
  We consider two biologically plausible structures, the Spiking Neural Network (SNN) and the self-attention mechanism. We propose a novel Spiking Self Attention (SSA) as well as a powerful framework, named Spiking Transformer (Spikformer)
  arXiv  Detail & Related papers  (2022-09-29T14:16:49Z)
• Training High-Performance Low-Latency Spiking Neural Networks by Differentiation on Spike Representation [70.75043144299168]
  Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware. It is a challenge to efficiently train SNNs due to their non-differentiability. We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
  arXiv  Detail & Related papers  (2022-05-01T12:44:49Z)
• Spiking CapsNet: A Spiking Neural Network With A Biologically Plausible Routing Rule Between Capsules [9.658836348699161]
  Spiking neural network (SNN) has attracted much attention due to their powerful-temporal information representation ability. CapsNet does well in assembling and coupling different levels. We propose Spiking CapsNet by introducing the capsules into the modelling of neural networks.
  arXiv  Detail & Related papers  (2021-11-15T14:23:15Z)
• Fully Spiking Variational Autoencoder [66.58310094608002]
  Spiking neural networks (SNNs) can be run on neuromorphic devices with ultra-high speed and ultra-low energy consumption. In this study, we build a variational autoencoder (VAE) with SNN to enable image generation.
  arXiv  Detail & Related papers  (2021-09-26T06:10:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.