Spiking Local Interaction and Adaptive Complementary Fusion for Spiking Transformer

• URL: http://arxiv.org/abs/2608.19238v1
• Date: Tue, 11 Aug 2026 14:23:26 GMT
• Title: Spiking Local Interaction and Adaptive Complementary Fusion for Spiking Transformer
• Abstract summary: Spiking Transformers model token interactions primarily through spiking self-attention (SSA)<n>We introduce Spiking Local Interaction (SLI) and Adaptive Complementary Fusion (ACF)<n>ACF integrates SSA and SLI through layer-specific, channel-wise coefficients that adaptively balance their contributions at different network depths.
• Score: 19.12698354620558
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spiking Transformers model token interactions primarily through spiking self-attention (SSA). However, binary query and key representations map continuous similarities to sparse and discrete relation responses, which may suppress weak relations and limit the propagation of local spatial context. To address this limitation, we introduce Spiking Local Interaction (SLI) and Adaptive Complementary Fusion (ACF). SLI establishes an attention-independent pathway for direct information exchange among neighboring spiking tokens using lightweight depthwise--pointwise transformations. ACF integrates SSA and SLI through layer-specific, channel-wise coefficients that adaptively balance their contributions at different network depths. The proposed design preserves the original attention formulation and can be incorporated into different Spiking Transformer architectures with modest parameter overhead. Experiments on ImageNet-1K, CIFAR-10, CIFAR-100, CIFAR10-DVS, and ADE20K show consistent improvements across image classification, event-based recognition, and semantic segmentation. In particular, QKFormer with SLI and ACF achieves $84.37\%$ Top-1 accuracy on ImageNet-1K and $37.5\%$ mIoU on ADE20K, where the segmentation model is trained without ImageNet pretraining. Ablation studies and qualitative analyses further indicate that SSA and SLI capture complementary interaction patterns and that learnable fusion consistently outperforms fixed weighting.

Related papers

• TailProp: content-adaptive light- and heavy-tailed propagation for vision [11.825348402205051]
  We introduce TailProp, a hierarchical vision backbone for cross-regime adaptive propagation.<n>Across image classification, object detection, semantic segmentation, robustness, and cross-backbone restoration, TailProp consistently outperforms matched propagation baselines.
  arXiv  Detail & Related papers  (2026-09-10T04:38:45Z)
• Rethinking Attention Locality in Spiking Transformers [50.00501275379363]
  Spiking Transformers provide a promising paradigm for efficient visual processing with spike-driven computation.<n>Their Softmax-free Spiking Self-Attention (SSA) struggles to establish spatially localized token interactions.<n>We propose Spatially Contiguous Local Attention with Boundary Continuity Pathway (SCLA-BCP)<n>Our approach improves mAP@50 by up to 9.50% on COCO on 2017 and mIoU by up to 3.42% on ADE20K.
  arXiv  Detail & Related papers  (2026-08-09T07:36:17Z)
• SPARE: Structural Parameter-Free Affinity Regularization for Flow Matching [30.43836162456455]
  Regularizing internal representations has emerged as an effective accelerator, yet existing methods split into two families with complementary costs.<n>We propose Structural.<n>free Affinity Regularization (SPARE), a regularizer that matches the pairwise affinities of intermediate tokens to those of the clean latents.<n>On ImageNet $256 times 256$ with SiT backbones under matched 400K-iteration budgets, SPARE adds no encoder, head, or parameters and only 0.08 GB of training memory.
  arXiv  Detail & Related papers  (2026-08-03T09:50:51Z)
• SSR: Similarity-Shift Refinement for Training-Free Object-Centric Masks [0.6302369456012739]
  Similarity-Shift Refinement improves object-centric masks with a frozen self-supervised Vision Transformer.<n> SSR measures changes in pairwise patch similarity before and after self-attention value aggregation.<n>It propagates the initial soft slot assignments in a single refinement step, without retraining or modifying either model.
  arXiv  Detail & Related papers  (2026-08-02T09:06:06Z)
• Cross-Sample Relational Fusion: Unifying Domain Generalization and Class-Incremental Learning [83.67663266193308]
  Class-Incremental Learning (CIL) requires a learning system to learn new classes while retaining previously learned knowledge.<n>In real-world scenarios such as autonomous driving, a system trained on urban roads in sunny weather may later need to operate in rural or highway environments.<n>We propose a unified framework to address domain shift and catastrophic forgetting simultaneously.
  arXiv  Detail & Related papers  (2026-05-09T09:49:13Z)
• Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping [61.459927600301654]
  Multi-condition control is bottlenecked by the conventional concatenate-and-attend'' strategy.<n>Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant.<n>We propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies.
  arXiv  Detail & Related papers  (2026-02-06T16:39:10Z)
• FCC: Fully Connected Correlation for Few-Shot Segmentation [11.277022867553658]
  Few-shot segmentation (FSS) aims to segment the target object in a query image using only a small set of support images and masks. Previous methods have tried to obtain prior information by creating correlation maps from pixel-level correlation on final-layer or same-layer features. We introduce FCC (Fully Connected Correlation) to integrate pixel-level correlations between support and query features.
  arXiv  Detail & Related papers  (2024-11-18T03:32:02Z)
• Affine-based Deformable Attention and Selective Fusion for Semi-dense Matching [30.272791354494373]
  We introduce affine-based local attention to model cross-view deformations. We also present selective fusion to merge local and global messages from cross attention.
  arXiv  Detail & Related papers  (2024-05-22T17:57:37Z)
• Rotated Multi-Scale Interaction Network for Referring Remote Sensing Image Segmentation [63.15257949821558]
  Referring Remote Sensing Image (RRSIS) is a new challenge that combines computer vision and natural language processing. Traditional Referring Image (RIS) approaches have been impeded by the complex spatial scales and orientations found in aerial imagery. We introduce the Rotated Multi-Scale Interaction Network (RMSIN), an innovative approach designed for the unique demands of RRSIS.
  arXiv  Detail & Related papers  (2023-12-19T08:14:14Z)
• Adaptive Split-Fusion Transformer [90.04885335911729]
  We propose an Adaptive Split-Fusion Transformer (ASF-former) to treat convolutional and attention branches differently with adaptive weights. Experiments on standard benchmarks, such as ImageNet-1K, show that our ASF-former outperforms its CNN, transformer counterparts, and hybrid pilots in terms of accuracy.
  arXiv  Detail & Related papers  (2022-04-26T10:00:28Z)
• Semantic Correspondence with Transformers [68.37049687360705]
  We propose Cost Aggregation with Transformers (CATs) to find dense correspondences between semantically similar images. We include appearance affinity modelling to disambiguate the initial correlation maps and multi-level aggregation. We conduct experiments to demonstrate the effectiveness of the proposed model over the latest methods and provide extensive ablation studies.
  arXiv  Detail & Related papers  (2021-06-04T14:39:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.