Vision Transformer with Sparse Scan Prior

• URL: http://arxiv.org/abs/2405.13335v1
• Date: Wed, 22 May 2024 04:34:36 GMT
• Title: Vision Transformer with Sparse Scan Prior
• Authors: Qihang Fan, Huaibo Huang, Mingrui Chen, Ran He,
• Abstract summary: Inspired by the human eye's sparse scanning mechanism, we propose a textbfSparse textbfScan textbfSelf-textbfAttention mechanism. This mechanism predefines a series of Anchors of Interest for each token and employs local attention to efficiently model the spatial information around these anchors. Building on $rmS3rmA$, we introduce the textbfSparse textbfScan textbfVision
• Score: 57.37893387775829
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In recent years, Transformers have achieved remarkable progress in computer vision tasks. However, their global modeling often comes with substantial computational overhead, in stark contrast to the human eye's efficient information processing. Inspired by the human eye's sparse scanning mechanism, we propose a \textbf{S}parse \textbf{S}can \textbf{S}elf-\textbf{A}ttention mechanism ($\rm{S}^3\rm{A}$). This mechanism predefines a series of Anchors of Interest for each token and employs local attention to efficiently model the spatial information around these anchors, avoiding redundant global modeling and excessive focus on local information. This approach mirrors the human eye's functionality and significantly reduces the computational load of vision models. Building on $\rm{S}^3\rm{A}$, we introduce the \textbf{S}parse \textbf{S}can \textbf{Vi}sion \textbf{T}ransformer (SSViT). Extensive experiments demonstrate the outstanding performance of SSViT across a variety of tasks. Specifically, on ImageNet classification, without additional supervision or training data, SSViT achieves top-1 accuracies of \textbf{84.4\%/85.7\%} with \textbf{4.4G/18.2G} FLOPs. SSViT also excels in downstream tasks such as object detection, instance segmentation, and semantic segmentation. Its robustness is further validated across diverse datasets. Code will be available at \url{https://github.com/qhfan/SSViT}.

Related papers

• CAS-ViT: Convolutional Additive Self-attention Vision Transformers for Efficient Mobile Applications [59.193626019860226]
  Vision Transformers (ViTs) mark a revolutionary advance in neural networks with their token mixer's powerful global context capability. We introduce CAS-ViT: Convolutional Additive Self-attention Vision Transformers. We show that CAS-ViT achieves a competitive performance when compared to other state-of-the-art backbones.
  arXiv  Detail & Related papers  (2024-08-07T11:33:46Z)
• VisMin: Visual Minimal-Change Understanding [7.226130826257802]
  We introduce a new, challenging benchmark termed textbfVisual textbfMinimal-Change Understanding (VisMin) VisMin requires models to predict the correct image-caption match given two images and two captions. We generate a large-scale training dataset to finetune CLIP and Idefics2, showing significant improvements in fine-grained understanding across benchmarks.
  arXiv  Detail & Related papers  (2024-07-23T18:10:43Z)
• Learning Multi-view Anomaly Detection [42.94263165352097]
  This study explores the recently proposed challenging multi-view Anomaly Detection (AD) task. We introduce the textbfMulti-textbfView textbfAnomaly textbfDetection (textbfMVAD) framework, which learns and integrates features from multi-views.
  arXiv  Detail & Related papers  (2024-07-16T17:26:34Z)
• Semantic Equitable Clustering: A Simple, Fast and Effective Strategy for Vision Transformer [57.37893387775829]
  We introduce a fast and balanced clustering method, named textbfSemantic textbfEquitable textbfClustering (SEC) SEC clusters tokens based on their global semantic relevance in an efficient, straightforward manner. We propose a versatile vision backbone, SecViT, which attains an impressive textbf84.2% image classification accuracy with only textbf27M parameters and textbf4.4G FLOPs.
  arXiv  Detail & Related papers  (2024-05-22T04:49:00Z)
• RMT: Retentive Networks Meet Vision Transformers [59.827563438653975]
  Vision Transformer (ViT) has gained increasing attention in the computer vision community in recent years. Self-Attention lacks explicit spatial priors and bears a quadratic computational complexity. We propose RMT, a strong vision backbone with explicit spatial prior for general purposes.
  arXiv  Detail & Related papers  (2023-09-20T00:57:48Z)
• Vision Transformers: From Semantic Segmentation to Dense Prediction [139.15562023284187]
  We explore the global context learning potentials of vision transformers (ViTs) for dense visual prediction. Our motivation is that through learning global context at full receptive field layer by layer, ViTs may capture stronger long-range dependency information. We formulate a family of Hierarchical Local-Global (HLG) Transformers, characterized by local attention within windows and global-attention across windows in a pyramidal architecture.
  arXiv  Detail & Related papers  (2022-07-19T15:49:35Z)
• TopFormer: Token Pyramid Transformer for Mobile Semantic Segmentation [111.8342799044698]
  We present a mobile-friendly architecture named textbfToken textbfPyramid Vision Transtextbfformer (textbfTopFormer) The proposed textbfTopFormer takes Tokens from various scales as input to produce scale-aware semantic features, which are then injected into the corresponding tokens to augment the representation. On the ADE20K dataset, TopFormer achieves 5% higher accuracy in mIoU than MobileNetV3 with lower latency on an ARM-based mobile device.
  arXiv  Detail & Related papers  (2022-04-12T04:51:42Z)
• MUNet: Motion Uncertainty-aware Semi-supervised Video Object Segmentation [31.100954335785026]
  We advocate the return of the emphmotion information and propose a motion uncertainty-aware framework (MUNet) for semi-supervised video object segmentation. We introduce a motion-aware spatial attention module to effectively fuse the motion feature with the semantic feature. We achieve $76.5%$ $mathcalJ & mathcalF$ only using DAVIS17 for training, which significantly outperforms the textitSOTA methods under the low-data protocol.
  arXiv  Detail & Related papers  (2021-11-29T16:01:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.