Dilated Neighborhood Attention Transformer

• URL: http://arxiv.org/abs/2209.15001v1
• Date: Thu, 29 Sep 2022 17:57:08 GMT
• Title: Dilated Neighborhood Attention Transformer
• Abstract summary: We introduce Dilated Neighborhood Attention Transformer (DiNAT), a new hierarchical vision transformer. DiNAT variants enjoy significant improvements over attention-based baselines such as NAT and Swin. Our Large model is ahead of its Swin counterpart by 1.5% box AP in COCO object detection, 1.3% mask AP in COCO instance segmentation, and 1.1% mIoU in ADE20K semantic segmentation.
• Score: 16.35767262996978
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers are quickly becoming one of the most heavily applied deep learning architectures across modalities, domains, and tasks. In vision, on top of ongoing efforts into plain transformers, hierarchical transformers have also gained significant attention, thanks to their performance and easy integration into existing frameworks. These models typically employ localized attention mechanisms, such as the sliding-window Neighborhood Attention (NA) or Swin Transformer's Shifted Window Self Attention. While effective at reducing self attention's quadratic complexity, local attention weakens two of the most desirable properties of self attention: long range inter-dependency modeling, and global receptive field. In this paper, we introduce Dilated Neighborhood Attention (DiNA), a natural, flexible and efficient extension to NA that can capture more global context and expand receptive fields exponentially at no additional cost. NA's local attention and DiNA's sparse global attention complement each other, and therefore we introduce Dilated Neighborhood Attention Transformer (DiNAT), a new hierarchical vision transformer built upon both. DiNAT variants enjoy significant improvements over attention-based baselines such as NAT and Swin, as well as modern convolutional baseline ConvNeXt. Our Large model is ahead of its Swin counterpart by 1.5% box AP in COCO object detection, 1.3% mask AP in COCO instance segmentation, and 1.1% mIoU in ADE20K semantic segmentation, and faster in throughput. We believe combinations of NA and DiNA have the potential to empower various tasks beyond those presented in this paper. To support and encourage research in this direction, in vision and beyond, we open-source our project at: https://github.com/SHI-Labs/Neighborhood-Attention-Transformer.

Related papers

• ACC-ViT : Atrous Convolution's Comeback in Vision Transformers [5.224344210588584]
  We introduce Atrous Attention, a fusion of regional and sparse attention, which can adaptively consolidate both local and global information. We also propose a general vision transformer backbone, named ACC-ViT, following conventional practices for standard vision tasks. ACC-ViT is therefore a strong vision backbone, which is also competitive in mobile-scale versions, ideal for niche applications with small datasets.
  arXiv  Detail & Related papers  (2024-03-07T04:05:16Z)
• Exploring Consistency in Cross-Domain Transformer for Domain Adaptive Semantic Segmentation [51.10389829070684]
  Domain gap can cause discrepancies in self-attention. Due to this gap, the transformer attends to spurious regions or pixels, which deteriorates accuracy on the target domain. We propose adaptation on attention maps with cross-domain attention layers.
  arXiv  Detail & Related papers  (2022-11-27T02:40:33Z)
• Efficient Image Generation with Variadic Attention Heads [66.9694645123474]
  We propose a simple, yet powerful method to allow the attention heads of a single transformer to attend to multiple receptive fields.<n>We demonstrate our method utilizing Neighborhood Attention (NA) and integrate it into a StyleGAN based architecture for image generation.<n>With this work, dubbed StyleNAT, we are able to achieve a FID of 2.05 on FFHQ, a 6% improvement over StyleGAN-XL.
  arXiv  Detail & Related papers  (2022-11-10T18:55:48Z)
• MAFormer: A Transformer Network with Multi-scale Attention Fusion for Visual Recognition [45.68567088645708]
  We introduce Multi-scale Attention Fusion into transformer (MAFormer) MAFormer explores local aggregation and global feature extraction in a dual-stream framework for visual recognition. Our MAFormer achieves state-of-the-art performance on common vision tasks.
  arXiv  Detail & Related papers  (2022-08-31T06:29:27Z)
• Neighborhood Attention Transformer [24.478343522484604]
  We present Neighborhood Attention (NA), the first efficient and scalable sliding-window attention mechanism for vision. NA is a pixel-wise operation, localizing self attention (SA) to the nearest neighboring pixels. We develop NATTENNeighborhood Attention Extension, which allows NA to run up to 40% faster than Swin's WSA.
  arXiv  Detail & Related papers  (2022-04-14T17:55:15Z)
• BOAT: Bilateral Local Attention Vision Transformer [70.32810772368151]
  Early Vision Transformers such as ViT and DeiT adopt global self-attention, which is computationally expensive when the number of patches is large. Recent Vision Transformers adopt local self-attention mechanisms, where self-attention is computed within local windows. We propose a Bilateral lOcal Attention vision Transformer (BOAT), which integrates feature-space local attention with image-space local attention.
  arXiv  Detail & Related papers  (2022-01-31T07:09:50Z)
• Locally Shifted Attention With Early Global Integration [93.5766619842226]
  We propose an approach that allows for coarse global interactions and fine-grained local interactions already at early layers of a vision transformer. Our method is shown to be superior to both convolutional and transformer-based methods for image classification on CIFAR10, CIFAR100, and ImageNet.
  arXiv  Detail & Related papers  (2021-12-09T18:12:24Z)
• Focal Self-attention for Local-Global Interactions in Vision Transformers [90.9169644436091]
  We present focal self-attention, a new mechanism that incorporates both fine-grained local and coarse-grained global interactions. With focal self-attention, we propose a new variant of Vision Transformer models, called Focal Transformer, which achieves superior performance over the state-of-the-art vision Transformers.
  arXiv  Detail & Related papers  (2021-07-01T17:56:09Z)
• LocalViT: Bringing Locality to Vision Transformers [132.42018183859483]
  locality is essential for images since it pertains to structures like lines, edges, shapes, and even objects. We add locality to vision transformers by introducing depth-wise convolution into the feed-forward network. This seemingly simple solution is inspired by the comparison between feed-forward networks and inverted residual blocks.
  arXiv  Detail & Related papers  (2021-04-12T17:59:22Z)
• OmniNet: Omnidirectional Representations from Transformers [49.23834374054286]
  This paper proposes Omnidirectional Representations from Transformers ( OmniNet) In OmniNet, instead of maintaining a strictly horizontal receptive field, each token is allowed to attend to all tokens in the entire network. Experiments are conducted on autoregressive language modeling, Machine Translation, Long Range Arena (LRA), and Image Recognition.
  arXiv  Detail & Related papers  (2021-03-01T15:31:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.