Fourier or Wavelet bases as counterpart self-attention in spikformer for efficient visual classification

• URL: http://arxiv.org/abs/2403.18228v1
• Date: Wed, 27 Mar 2024 03:31:16 GMT
• Title: Fourier or Wavelet bases as counterpart self-attention in spikformer for efficient visual classification
• Abstract summary: Energy-efficient spikformer has been proposed by integrating the biologically plausible spiking neural network (SNN) and artificial Transformer. In this paper, we innovatively replace vanilla SSA with spike-form Fourier Transform, Wavelet Transform, and their combinations, based on a key hypothesis that both of them use a set of basis functions for information transformation. The FWformer can achieve comparable or even higher accuracies ($0.4%$-$1.5%$), higher running speed ($9%$-$51%$ for training and $19%$-$70%$ for inference
• Score: 9.312432667108691
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Energy-efficient spikformer has been proposed by integrating the biologically plausible spiking neural network (SNN) and artificial Transformer, whereby the Spiking Self-Attention (SSA) is used to achieve both higher accuracy and lower computational cost. However, it seems that self-attention is not always necessary, especially in sparse spike-form calculation manners. In this paper, we innovatively replace vanilla SSA (using dynamic bases calculating from Query and Key) with spike-form Fourier Transform, Wavelet Transform, and their combinations (using fixed triangular or wavelets bases), based on a key hypothesis that both of them use a set of basis functions for information transformation. Hence, the Fourier-or-Wavelet-based spikformer (FWformer) is proposed and verified in visual classification tasks, including both static image and event-based video datasets. The FWformer can achieve comparable or even higher accuracies ($0.4\%$-$1.5\%$), higher running speed ($9\%$-$51\%$ for training and $19\%$-$70\%$ for inference), reduced theoretical energy consumption ($20\%$-$25\%$), and reduced GPU memory usage ($4\%$-$26\%$), compared to the standard spikformer. Our result indicates the continuous refinement of new Transformers, that are inspired either by biological discovery (spike-form), or information theory (Fourier or Wavelet Transform), is promising.

Related papers

• Circuit Fine-Tuning for Compute-Efficient Transformer Adaptation [3.52359746858894]
  Circuit Fine-Tuning (CFT) is a compute-efficient framework that uses circuit discovery to select modules for fine-tuning before training.<n>CFT needs no learning-rate warmup and reaches peak accuracy in $sim20$ epochs on average.
  arXiv  Detail & Related papers  (2026-08-08T21:04:19Z)
• The RG-Flow Transformer: Encoding Scale-Free Dynamics in Scarce EEG [0.09229852843814058]
  We ask whether a transformer endowed with an explicit renormalization-group inductive bias the RG-Flow Transformer has an advantage over a parameter-matched vanilla transformer on emphreal, scarce EEG.<n>We benchmark RG-Flow against a param-matched vanilla transformer and a hierarchy-only ablation on 5-class AASM sleep staging.<n>Across $5$ subjects and $5$ seeds under leave-one-subject-out cross-validation, RG-Flow and the vanilla transformer are statistically indistinguishable on 5-class staging.
  arXiv  Detail & Related papers  (2026-07-11T18:49:13Z)
• Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes [53.726365933748134]
  We analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions.<n>We characterize the sparsity-accuracy tradeoff across 79 configurations and identify a sharp accuracy cliff above $sim$70% activation sparsity.
  arXiv  Detail & Related papers  (2026-05-17T21:29:20Z)
• Closing the Theory-Practice Gap in Spiking Transformers via Effective Dimension [13.891522069967507]
  Spiking transformers achieve competitive accuracy with conventional transformers while offering $38$-$57times$ energy efficiency on neuromorphic hardware.<n>This paper establishes the first comprehensive expressivity theory for spiking self-attention.
  arXiv  Detail & Related papers  (2026-04-17T07:15:53Z)
• EquiformerV3: Scaling Efficient, Expressive, and General SE(3)-Equivariant Graph Attention Transformers [70.68282700439444]
  We introduce EquiformerV3, the third generation of the $SE(3)$-equivariant graph attention Transformer.<n>We optimize the software implementation, achieving $1.75times$ speedup.<n>We propose SwiGLU-$S2$ activations to incorporate many-body interactions for better theoretical expressivity.
  arXiv  Detail & Related papers  (2026-04-10T09:12:16Z)
• Scaling Diffusion Transformers Efficiently via $μ$P [61.64043787316389]
  Maximal Update Parametrization ($mu$P) was proposed for vanilla Transformers, which enables stable HP transfer from small to large language models.<n>We generalize standard $mu$P to diffusion Transformers and validate its effectiveness through large-scale experiments.
  arXiv  Detail & Related papers  (2025-05-21T08:49:03Z)
• On the Role of Depth and Looping for In-Context Learning with Task Diversity [69.4145579827826]
  We study in-context learning for linear regression with diverse tasks. We show that multilayer Transformers are not robust to even distributional shifts as small as $O(e-L)$ in Wasserstein distance.
  arXiv  Detail & Related papers  (2024-10-29T03:27:56Z)
• Spiking Wavelet Transformer [1.8712213089437697]
  Spiking neural networks (SNNs) offer an energy-efficient alternative to conventional deep learning. Transformers with SNNs have shown promise for accuracy, but struggle to learn high-frequency patterns. We propose the Spiking Wavelet Transformer (SWformer), an attention-free architecture that effectively learns comprehensive spatial-frequency features in a spike-driven manner.
  arXiv  Detail & Related papers  (2024-03-17T08:41:48Z)
• p-Laplacian Transformer [7.2541371193810384]
  $p$-Laplacian regularization, rooted in graph and image signal processing, introduces a parameter $p$ to control the regularization effect on these data. We first show that the self-attention mechanism obtains the minimal Laplacian regularization. We then propose a novel class of transformers, namely the $p$-Laplacian Transformer (p-LaT)
  arXiv  Detail & Related papers  (2023-11-06T16:25:56Z)
• Attention-free Spikformer: Mixing Spike Sequences with Simple Linear Transforms [16.54314950692779]
  Spikformer integrates self-attention capability and the biological properties of Spiking Neural Networks (SNNs) It introduces a Spiking Self-Attention (SSA) module to mix sparse visual features using spike-form Query, Key, and Value. We conduct extensive experiments on image classification using both neuromorphic and static datasets.
  arXiv  Detail & Related papers  (2023-08-02T11:41:54Z)
• Spike-driven Transformer [31.931401322707995]
  Spiking Neural Networks (SNNs) provide an energy-efficient deep learning option due to their unique spike-based event-driven (i.e., spike-driven) paradigm. In this paper, we incorporate the spike-driven paradigm into Transformer by the proposed Spike-driven Transformer with four unique properties. It is shown that the Spike-driven Transformer can achieve 77.1% top-1 accuracy on ImageNet-1K, which is the state-of-the-art result in the SNN field.
  arXiv  Detail & Related papers  (2023-07-04T13:00:18Z)
• Spikformer: When Spiking Neural Network Meets Transformer [102.91330530210037]
  We consider two biologically plausible structures, the Spiking Neural Network (SNN) and the self-attention mechanism. We propose a novel Spiking Self Attention (SSA) as well as a powerful framework, named Spiking Transformer (Spikformer)
  arXiv  Detail & Related papers  (2022-09-29T14:16:49Z)
• Wave-ViT: Unifying Wavelet and Transformers for Visual Representation Learning [138.29273453811945]
  Multi-scale Vision Transformer (ViT) has emerged as a powerful backbone for computer vision tasks. We propose a new Wavelet Vision Transformer (textbfWave-ViT) that formulates the invertible down-sampling with wavelet transforms and self-attention learning.
  arXiv  Detail & Related papers  (2022-07-11T16:03:51Z)
• Your Transformer May Not be as Powerful as You Expect [88.11364619182773]
  We mathematically analyze the power of RPE-based Transformers regarding whether the model is capable of approximating any continuous sequence-to-sequence functions. We present a negative result by showing there exist continuous sequence-to-sequence functions that RPE-based Transformers cannot approximate no matter how deep and wide the neural network is. We develop a novel attention module, called Universal RPE-based (URPE) Attention, which satisfies the conditions.
  arXiv  Detail & Related papers  (2022-05-26T14:51:30Z)
• Adaptive Split-Fusion Transformer [90.04885335911729]
  We propose an Adaptive Split-Fusion Transformer (ASF-former) to treat convolutional and attention branches differently with adaptive weights. Experiments on standard benchmarks, such as ImageNet-1K, show that our ASF-former outperforms its CNN, transformer counterparts, and hybrid pilots in terms of accuracy.
  arXiv  Detail & Related papers  (2022-04-26T10:00:28Z)
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers [71.31712741938837]
  We show that sparse Transformers with only $O(n)$ connections per attention layer can approximate the same function class as the dense model with $n2$ connections. We also present experiments comparing different patterns/levels of sparsity on standard NLP tasks.
  arXiv  Detail & Related papers  (2020-06-08T18:30:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.