ViTAR: Vision Transformer with Any Resolution

• URL: http://arxiv.org/abs/2403.18361v2
• Date: Thu, 28 Mar 2024 14:59:44 GMT
• Title: ViTAR: Vision Transformer with Any Resolution
• Authors: Qihang Fan, Quanzeng You, Xiaotian Han, Yongfei Liu, Yunzhe Tao, Huaibo Huang, Ran He, Hongxia Yang,
• Abstract summary: Vision Transformers experience a performance decline when processing resolutions different from those seen during training. We introduce fuzzy positional encoding in the Vision Transformer to provide consistent positional awareness across multiple resolutions. Our resulting model, ViTAR, demonstrates impressive adaptability, achieving 83.3% top-1 accuracy at a 1120x1120 resolution and 80.4% accuracy at a 4032x4032 resolution.
• Score: 80.95324692984903
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper tackles a significant challenge faced by Vision Transformers (ViTs): their constrained scalability across different image resolutions. Typically, ViTs experience a performance decline when processing resolutions different from those seen during training. Our work introduces two key innovations to address this issue. Firstly, we propose a novel module for dynamic resolution adjustment, designed with a single Transformer block, specifically to achieve highly efficient incremental token integration. Secondly, we introduce fuzzy positional encoding in the Vision Transformer to provide consistent positional awareness across multiple resolutions, thereby preventing overfitting to any single training resolution. Our resulting model, ViTAR (Vision Transformer with Any Resolution), demonstrates impressive adaptability, achieving 83.3\% top-1 accuracy at a 1120x1120 resolution and 80.4\% accuracy at a 4032x4032 resolution, all while reducing computational costs. ViTAR also shows strong performance in downstream tasks such as instance and semantic segmentation and can easily combined with self-supervised learning techniques like Masked AutoEncoder. Our work provides a cost-effective solution for enhancing the resolution scalability of ViTs, paving the way for more versatile and efficient high-resolution image processing.

Related papers

• MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution [31.564277546050484]
  We propose to enhance the model adaptability to resolution variation by optimizing the patch embedding. The proposed method, called Multi-Scale Patch Embedding (MSPE), substitutes the standard patch embedding with multiple variable-sized patch kernels. Our method does not require high-cost training or modifications to other parts, making it easy to apply to most ViT models.
  arXiv  Detail & Related papers  (2024-05-28T14:50:12Z)
• DehazeDCT: Towards Effective Non-Homogeneous Dehazing via Deformable Convolutional Transformer [43.807338032286346]
  We introduce an innovative non-homogeneous Dehazing method via Deformable Convolutional Transformer-like architecture (DehazeDCT) We first design a transformer-like network based on deformable convolution v4, which offers long-range dependency and adaptive spatial aggregation capabilities. Furthermore, we leverage a lightweight Retinex-inspired transformer to achieve color correction and structure refinement.
  arXiv  Detail & Related papers  (2024-05-24T10:59:18Z)
• HIRI-ViT: Scaling Vision Transformer with High Resolution Inputs [102.4965532024391]
  hybrid deep models of Vision Transformer (ViT) and Convolution Neural Network (CNN) have emerged as a powerful class of backbones for vision tasks. We present a new hybrid backbone with HIgh-Resolution Inputs (namely HIRI-ViT), that upgrades prevalent four-stage ViT to five-stage ViT tailored for high-resolution inputs. HiRI-ViT achieves to-date the best published Top-1 accuracy of 84.3% on ImageNet with 448$times$448 inputs, which absolutely improves 83.4% of iFormer-S by 0.9% with 224$times
  arXiv  Detail & Related papers  (2024-03-18T17:34:29Z)
• ResFormer: Scaling ViTs with Multi-Resolution Training [100.01406895070693]
  We introduce ResFormer, a framework for improved performance on a wide spectrum of, mostly unseen, testing resolutions. In particular, ResFormer operates on replicated images of different resolutions and enforces a scale consistency loss to engage interactive information across different scales. We demonstrate, moreover, ResFormer is flexible and can be easily extended to semantic segmentation, object detection and video action recognition.
  arXiv  Detail & Related papers  (2022-12-01T18:57:20Z)
• Vicinity Vision Transformer [53.43198716947792]
  We present a Vicinity Attention that introduces a locality bias to vision transformers with linear complexity. Our approach achieves state-of-the-art image classification accuracy with 50% fewer parameters than previous methods.
  arXiv  Detail & Related papers  (2022-06-21T17:33:53Z)
• AdaViT: Adaptive Tokens for Efficient Vision Transformer [91.88404546243113]
  We introduce AdaViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. AdaViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds.
  arXiv  Detail & Related papers  (2021-12-14T18:56:07Z)
• Swin Transformer V2: Scaling Up Capacity and Resolution [45.462916348268664]
  We present techniques for scaling Swin Transformer up to 3 billion parameters and making it capable of training with images of up to 1,536$times$1,536 resolution. By scaling up capacity and resolution, Swin Transformer sets new records on four representative vision benchmarks.
  arXiv  Detail & Related papers  (2021-11-18T18:59:33Z)
• HRFormer: High-Resolution Transformer for Dense Prediction [99.6060997466614]
  We present a High-Resolution Transformer (HRFormer) that learns high-resolution representations for dense prediction tasks. We take advantage of the multi-resolution parallel design introduced in high-resolution convolutional networks (HRNet) We demonstrate the effectiveness of the High-Resolution Transformer on both human pose estimation and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2021-10-18T15:37:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.