Neighborhood Attention Transformer

• URL: http://arxiv.org/abs/2204.07143v5
• Date: Tue, 16 May 2023 21:26:30 GMT
• Title: Neighborhood Attention Transformer
• Abstract summary: We present Neighborhood Attention (NA), the first efficient and scalable sliding-window attention mechanism for vision. NA is a pixel-wise operation, localizing self attention (SA) to the nearest neighboring pixels. We develop NATTENNeighborhood Attention Extension, which allows NA to run up to 40% faster than Swin's WSA.
• Score: 24.478343522484604
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present Neighborhood Attention (NA), the first efficient and scalable sliding-window attention mechanism for vision. NA is a pixel-wise operation, localizing self attention (SA) to the nearest neighboring pixels, and therefore enjoys a linear time and space complexity compared to the quadratic complexity of SA. The sliding-window pattern allows NA's receptive field to grow without needing extra pixel shifts, and preserves translational equivariance, unlike Swin Transformer's Window Self Attention (WSA). We develop NATTEN (Neighborhood Attention Extension), a Python package with efficient C++ and CUDA kernels, which allows NA to run up to 40% faster than Swin's WSA while using up to 25% less memory. We further present Neighborhood Attention Transformer (NAT), a new hierarchical transformer design based on NA that boosts image classification and downstream vision performance. Experimental results on NAT are competitive; NAT-Tiny reaches 83.2% top-1 accuracy on ImageNet, 51.4% mAP on MS-COCO and 48.4% mIoU on ADE20K, which is 1.9% ImageNet accuracy, 1.0% COCO mAP, and 2.6% ADE20K mIoU improvement over a Swin model with similar size. To support more research based on sliding-window attention, we open source our project and release our checkpoints at: https://github.com/SHI-Labs/Neighborhood-Attention-Transformer .

Related papers

• CoSwin: Convolution Enhanced Hierarchical Shifted Window Attention For Small-Scale Vision [2.558238597112103]
  CoSwin is a novel feature-fusion architecture that augments the hierarchical shifted window attention with localized convolutional feature learning.<n>We evaluate CoSwin on multiple image classification benchmarks including CIFAR-10, CIFAR-100, MNIST, SVHN, and Tiny ImageNet.
  arXiv  Detail & Related papers  (2025-09-10T19:43:16Z)
• ZoomTrack: Target-aware Non-uniform Resizing for Efficient Visual Tracking [40.13014036490452]
  transformer has enabled the speed-oriented trackers to approach state-of-the-art (SOTA) performance with high-speed. We demonstrate that it is possible to narrow or even close this gap while achieving high tracking speed based on the smaller input size.
  arXiv  Detail & Related papers  (2023-10-16T05:06:13Z)
• DilateFormer: Multi-Scale Dilated Transformer for Visual Recognition [62.95223898214866]
  We explore effective Vision Transformers to pursue a preferable trade-off between the computational complexity and size of the attended receptive field. With a pyramid architecture, we construct a Multi-Scale Dilated Transformer (DilateFormer) by stacking MSDA blocks at low-level stages and global multi-head self-attention blocks at high-level stages. Our experiment results show that our DilateFormer achieves state-of-the-art performance on various vision tasks.
  arXiv  Detail & Related papers  (2023-02-03T14:59:31Z)
• Vision Transformer with Super Token Sampling [93.70963123497327]
  Vision transformer has achieved impressive performance for many vision tasks. It may suffer from high redundancy in capturing local features for shallow layers. Super tokens attempt to provide a semantically meaningful tessellation of visual content.
  arXiv  Detail & Related papers  (2022-11-21T03:48:13Z)
• Efficient Image Generation with Variadic Attention Heads [66.9694645123474]
  We propose a simple, yet powerful method to allow the attention heads of a single transformer to attend to multiple receptive fields.<n>We demonstrate our method utilizing Neighborhood Attention (NA) and integrate it into a StyleGAN based architecture for image generation.<n>With this work, dubbed StyleNAT, we are able to achieve a FID of 2.05 on FFHQ, a 6% improvement over StyleGAN-XL.
  arXiv  Detail & Related papers  (2022-11-10T18:55:48Z)
• Dilated Neighborhood Attention Transformer [16.35767262996978]
  We introduce Dilated Neighborhood Attention Transformer (DiNAT), a new hierarchical vision transformer. DiNAT variants enjoy significant improvements over attention-based baselines such as NAT and Swin. Our Large model is ahead of its Swin counterpart by 1.5% box AP in COCO object detection, 1.3% mask AP in COCO instance segmentation, and 1.1% mIoU in ADE20K semantic segmentation.
  arXiv  Detail & Related papers  (2022-09-29T17:57:08Z)
• Vicinity Vision Transformer [53.43198716947792]
  We present a Vicinity Attention that introduces a locality bias to vision transformers with linear complexity. Our approach achieves state-of-the-art image classification accuracy with 50% fewer parameters than previous methods.
  arXiv  Detail & Related papers  (2022-06-21T17:33:53Z)
• Adaptive Split-Fusion Transformer [90.04885335911729]
  We propose an Adaptive Split-Fusion Transformer (ASF-former) to treat convolutional and attention branches differently with adaptive weights. Experiments on standard benchmarks, such as ImageNet-1K, show that our ASF-former outperforms its CNN, transformer counterparts, and hybrid pilots in terms of accuracy.
  arXiv  Detail & Related papers  (2022-04-26T10:00:28Z)
• SepViT: Separable Vision Transformer [20.403430632658946]
  Vision Transformers often rely on extensive computational costs to achieve high performance, which is burdensome to deploy on resource-constrained devices. We draw lessons from depthwise separable convolution and imitate its ideology to design an efficient Transformer backbone, i.e., Separable Vision Transformer, abbreviated as SepViT. SepViT helps to carry out the local-global information interaction within and among the windows in sequential order via a depthwise separable self-attention.
  arXiv  Detail & Related papers  (2022-03-29T09:20:01Z)
• UniFormer: Unifying Convolution and Self-attention for Visual Recognition [69.68907941116127]
  Convolution neural networks (CNNs) and vision transformers (ViTs) have been two dominant frameworks in the past few years. We propose a novel Unified transFormer (UniFormer) which seamlessly integrates the merits of convolution and self-attention in a concise transformer format. Our UniFormer achieves 86.3 top-1 accuracy on ImageNet-1K classification.
  arXiv  Detail & Related papers  (2022-01-24T04:39:39Z)
• CSWin Transformer: A General Vision Transformer Backbone with Cross-Shaped Windows [99.36226415086243]
  We present CSWin Transformer, an efficient and effective Transformer-based backbone for general-purpose vision tasks. A challenging issue in Transformer design is that global self-attention is very expensive to compute whereas local self-attention often limits the field of interactions of each token.
  arXiv  Detail & Related papers  (2021-07-01T17:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.