Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention

• URL: http://arxiv.org/abs/2605.13887v1
• Date: Tue, 12 May 2026 02:08:08 GMT
• Title: Breaking Global Self-Attention Bottlenecks in Transformer-based Spiking Neural Networks with Local Structure-Aware Self-Attention
• Abstract summary: Spiking Neural Networks (SNNs) integrate SNNs with global self-attention.<n>Existing Transformer-based SNNs suffer from two fundamental limitations.<n>We develop Local Structure-Aware Spiking Transformer (LSFormer)
• Score: 2.098476785894547
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformer-based Spiking Neural Networks (SNNs) integrate SNNs with global self-attention and have demonstrated impressive performance. However, existing Transformer-based SNNs suffer from two fundamental limitations. First, they typically employ max pooling layers to reduce the size of feature maps, but the max pooling captures only the strongest response and fails to comprehensively preserve representative regional features. Second, the global self-attention involves all global feature interactions, resulting in computational redundancy and quadratic computational complexity, thus conflicting with the sparse and energy-efficient characteristics of SNNs. To address these challenges, we develop Local Structure-Aware Spiking Transformer (LSFormer), a novel Transformer-based Spiking Neural Network that incorporates Spiking Response Pooling (SPooling) and Local Structure-Aware Spiking Self-Attention (LS-SSA). For the first time, our LSFormer leverages a local dilated window mechanism to capture both local details and long-range dependencies. Experimental results demonstrate that our LSFormer achieves state-of-the-art performance compared to existing advanced Transformer-based SNNs. Notably, on the more challenging static dataset Tiny-ImageNet and neuromorphic dataset N-CALTECH101, LSFormer substantially outperforms state-of-the-art baselines by 4.3\% and 8.6\% in top-1 classification accuracy, respectively. These results highlight the potential of LSFormer to advance energy-efficient spiking models toward practical deployment in large-scale vision applications.

Related papers

• Reducing ANN-SNN Conversion Error via Residual Membrane Potential Alignment [52.910770415431244]
  Spiking Neural Networks (SNNs) serve as core architectures for neuromorphic computing.<n>ANN-SNN conversion circumvents issues by reusing well-trained ANN weights for low-latency, energy-efficient inference.<n>Existing conversion schemes suffer from severe accuracy drops at small timesteps, large inference delays and cumulative quantization errors.
  arXiv  Detail & Related papers  (2026-08-14T04:46:03Z)
• Elastic Spiking Transformers for Efficient Gesture Understanding [6.52225846892392]
  Spiking Neural Networks (SNNs) offer energy-efficient processing of event-based sensor data for healthcare applications.<n>We introduce the Elastic Spiking Transformer, a runtime-adaptive architecture that brings elasticity into the spiking paradigm.<n>We evaluate the approach on CIFAR10/100, CIFAR10-DVS, and the EHWGesture clinical gesture understanding dataset.
  arXiv  Detail & Related papers  (2026-05-04T12:35:52Z)
• Ge$^\text{2}$mS-T: Multi-Dimensional Grouping for Ultra-High Energy Efficiency in Spiking Transformer [84.8831358775386]
  Spiking Neural Networks (SNNs) offer superior energy efficiency over Artificial Neural Networks (ANNs)<n>Existing paradigms including ANN-SNN Conversion and Spatial-Temporal Backpropagation (STBP) suffer from inherent limitations.<n>We propose Ge$text2$mS-T, a novel architecture implementing grouped computation across temporal, spatial and network structure dimensions.
  arXiv  Detail & Related papers  (2026-04-10T02:58:46Z)
• Neural Dynamics Self-Attention for Spiking Transformers [59.57278823240336]
  Spiking Neural Networks (SNNs) with Transformer architectures offer a promising pathway to balance energy efficiency and performance.<n>Existing Spiking Transformers face two critical challenges: (i) a substantial performance gap compared to their Artificial Neural Networks (ANNs) counterparts and (ii) high memory overhead during inference.<n>We propose LRF-Dyn, which uses spiking neurons with localized receptive fields to compute attention while reducing memory requirements.
  arXiv  Detail & Related papers  (2026-03-09T12:23:37Z)
• EUGens: Efficient, Unified, and General Dense Layers [56.498769704575544]
  We propose a new class of dense layers that generalize standard fully-connected feedforward layers, textbfEfficient, textbfUnimat and textbfGeneral dense layers (EUGens)<n>EUGens leverage random features to approximate standard FFLs and go beyond them by incorporating a direct dependence on the input norms in their computations.
  arXiv  Detail & Related papers  (2026-01-30T05:01:03Z)
• Self-cross Feature based Spiking Neural Networks for Efficient Few-shot Learning [16.156610945877986]
  We propose a few-shot learning framework based on Spiking Neural Networks (SNNs)<n>We apply the combination of temporal efficient training loss and Info Info loss to optimize the temporal spike dynamics of trains and enhance the discriminative power.
  arXiv  Detail & Related papers  (2025-05-12T16:51:08Z)
• ScaleGNN: Towards Scalable Graph Neural Networks via Adaptive High-order Neighboring Feature Fusion [73.85920403511706]
  We propose ScaleGNN, a novel framework that adaptively fuses multi-hop node features for scalable and effective graph learning.<n>We show that ScaleGNN consistently outperforms state-of-the-art GNNs in both predictive accuracy and computational efficiency.
  arXiv  Detail & Related papers  (2025-04-22T14:05:11Z)
• Unifying Dimensions: A Linear Adaptive Approach to Lightweight Image Super-Resolution [6.857919231112562]
  Window-based transformers have demonstrated outstanding performance in super-resolution tasks. They exhibit higher computational complexity and inference latency than convolutional neural networks. We construct a convolution-based Transformer framework named the linear adaptive mixer network (LAMNet)
  arXiv  Detail & Related papers  (2024-09-26T07:24:09Z)
• AutoST: Training-free Neural Architecture Search for Spiking Transformers [14.791412391584064]
  Spiking Transformers achieve both the energy efficiency of Spiking Neural Networks (SNNs) and the high capacity of Transformers. Existing Spiking Transformer architectures exhibit a notable architectural gap, resulting in suboptimal performance. We introduce AutoST, a training-free NAS method for Spiking Transformers, to rapidly identify high-performance Spiking Transformer architectures.
  arXiv  Detail & Related papers  (2023-07-01T10:19:52Z)
• Efficient Deep Spiking Multi-Layer Perceptrons with Multiplication-Free Inference [13.924924047051782]
  Deep convolution architectures for Spiking Neural Networks (SNNs) have significantly enhanced image classification performance and reduced computational burdens. This research explores a new pathway, drawing inspiration from the progress made in Multi-Layer Perceptrons (MLPs) We propose an innovative spiking architecture that uses batch normalization to retain MFI compatibility. We establish an efficient multi-stage spiking network that blends effectively global receptive fields with local feature extraction.
  arXiv  Detail & Related papers  (2023-06-21T16:52:20Z)
• Spikformer: When Spiking Neural Network Meets Transformer [102.91330530210037]
  We consider two biologically plausible structures, the Spiking Neural Network (SNN) and the self-attention mechanism. We propose a novel Spiking Self Attention (SSA) as well as a powerful framework, named Spiking Transformer (Spikformer)
  arXiv  Detail & Related papers  (2022-09-29T14:16:49Z)
• Cross-receptive Focused Inference Network for Lightweight Image Super-Resolution [64.25751738088015]
  Transformer-based methods have shown impressive performance in single image super-resolution (SISR) tasks. Transformers that need to incorporate contextual information to extract features dynamically are neglected. We propose a lightweight Cross-receptive Focused Inference Network (CFIN) that consists of a cascade of CT Blocks mixed with CNN and Transformer.
  arXiv  Detail & Related papers  (2022-07-06T16:32:29Z)
• Modeling from Features: a Mean-field Framework for Over-parameterized Deep Neural Networks [54.27962244835622]
  This paper proposes a new mean-field framework for over- parameterized deep neural networks (DNNs) In this framework, a DNN is represented by probability measures and functions over its features in the continuous limit. We illustrate the framework via the standard DNN and the Residual Network (Res-Net) architectures.
  arXiv  Detail & Related papers  (2020-07-03T01:37:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.