Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution

• URL: http://arxiv.org/abs/2307.06304v1
• Date: Wed, 12 Jul 2023 17:01:03 GMT
• Title: Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution
• Abstract summary: Vision Transformer (ViT) models offer flexible sequence-based modeling. Native Resolution ViT uses sequence packing during training to process inputs of arbitrary resolutions and aspect ratios. NaViT can be efficiently transferred to standard tasks such as image and video classification, object detection, and semantic segmentation.
• Score: 34.05028995319689
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The ubiquitous and demonstrably suboptimal choice of resizing images to a fixed resolution before processing them with computer vision models has not yet been successfully challenged. However, models such as the Vision Transformer (ViT) offer flexible sequence-based modeling, and hence varying input sequence lengths. We take advantage of this with NaViT (Native Resolution ViT) which uses sequence packing during training to process inputs of arbitrary resolutions and aspect ratios. Alongside flexible model usage, we demonstrate improved training efficiency for large-scale supervised and contrastive image-text pretraining. NaViT can be efficiently transferred to standard tasks such as image and video classification, object detection, and semantic segmentation and leads to improved results on robustness and fairness benchmarks. At inference time, the input resolution flexibility can be used to smoothly navigate the test-time cost-performance trade-off. We believe that NaViT marks a departure from the standard, CNN-designed, input and modelling pipeline used by most computer vision models, and represents a promising direction for ViTs.

Related papers

• TuringViT: Making SOTA Vision Transformers Accessible to All [40.44858252534576]
  TuringViT addresses challenges with three key designs: Turing Linear Attention, VISTA-Curation, and dynamic-resolution pretraining.<n>TuringViT outperforms leading open-source ViT baselines with only 10% of the data.<n>Our scaling-law analysis shows that TuringViT continues to improve predictably with curated data scale, far from saturation.
  arXiv  Detail & Related papers  (2026-06-23T07:42:22Z)
• FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model [76.84519526283083]
  We present the textbfFlexible Vision Transformer (FiT), a transformer architecture specifically designed for generating images with textitunrestricted resolutions and aspect ratios FiTv2 exhibits $2times$ convergence speed of FiT, when incorporating advanced training-free extrapolation techniques. Comprehensive experiments demonstrate the exceptional performance of FiTv2 across a broad range of resolutions.
  arXiv  Detail & Related papers  (2024-10-17T15:51:49Z)
• ViTAR: Vision Transformer with Any Resolution [80.95324692984903]
  Vision Transformers experience a performance decline when processing resolutions different from those seen during training. We introduce fuzzy positional encoding in the Vision Transformer to provide consistent positional awareness across multiple resolutions. Our resulting model, ViTAR, demonstrates impressive adaptability, achieving 83.3% top-1 accuracy at a 1120x1120 resolution and 80.4% accuracy at a 4032x4032 resolution.
  arXiv  Detail & Related papers  (2024-03-27T08:53:13Z)
• Denoising Vision Transformers [43.03068202384091]
  We propose a two-stage denoising approach, termed Denoising Vision Transformers (DVT) In the first stage, we separate the clean features from those contaminated by positional artifacts by enforcing cross-view feature consistency with neural fields on a per-image basis. In the second stage, we train a lightweight transformer block to predict clean features from raw ViT outputs, leveraging the derived estimates of the clean features as supervision.
  arXiv  Detail & Related papers  (2024-01-05T18:59:52Z)
• PriViT: Vision Transformers for Fast Private Inference [55.36478271911595]
  Vision Transformer (ViT) architecture has emerged as the backbone of choice for state-of-the-art deep models for computer vision applications. ViTs are ill-suited for private inference using secure multi-party protocols, due to the large number of non-polynomial operations. We propose PriViT, an algorithm to selectively " Taylorize" nonlinearities in ViTs while maintaining their prediction accuracy.
  arXiv  Detail & Related papers  (2023-10-06T21:45:05Z)
• TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection [61.0662744915659]
  We propose an efficient vision-and-language pre-training model with textbfText-textbfRelevant textbfImage textbfPatch textbfSelection, namely TRIPS.<n> TRIPS reduces the visual sequence progressively with a text-guided patch-selection layer in the visual backbone for efficient training and inference.
  arXiv  Detail & Related papers  (2023-05-08T05:53:30Z)
• Where are my Neighbors? Exploiting Patches Relations in Self-Supervised Vision Transformer [3.158346511479111]
  We propose a simple but still effective self-supervised learning (SSL) strategy to train Vision Transformers (ViTs) We define a set of SSL tasks based on relations of image patches that the model has to solve before or jointly during the downstream training. Our RelViT model optimize all the output tokens of the transformer encoder that are related to the image patches, thus exploiting more training signal at each training step.
  arXiv  Detail & Related papers  (2022-06-01T13:25:32Z)
• Coarse-to-Fine Vision Transformer [83.45020063642235]
  We propose a coarse-to-fine vision transformer (CF-ViT) to relieve computational burden while retaining performance. Our proposed CF-ViT is motivated by two important observations in modern ViT models. Our CF-ViT reduces 53% FLOPs of LV-ViT, and also achieves 2.01x throughput.
  arXiv  Detail & Related papers  (2022-03-08T02:57:49Z)
• Intriguing Properties of Vision Transformers [114.28522466830374]
  Vision transformers (ViT) have demonstrated impressive performance across various machine vision problems. We systematically study this question via an extensive set of experiments and comparisons with a high-performing convolutional neural network (CNN) We show effective features of ViTs are due to flexible receptive and dynamic fields possible via the self-attention mechanism.
  arXiv  Detail & Related papers  (2021-05-21T17:59:18Z)
• Visformer: The Vision-friendly Transformer [105.52122194322592]
  We propose a new architecture named Visformer, which is abbreviated from the Vision-friendly Transformer' With the same computational complexity, Visformer outperforms both the Transformer-based and convolution-based models in terms of ImageNet classification accuracy.
  arXiv  Detail & Related papers  (2021-04-26T13:13:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.