Boosting vision transformers for image retrieval

• URL: http://arxiv.org/abs/2210.11909v1
• Date: Fri, 21 Oct 2022 12:17:12 GMT
• Title: Boosting vision transformers for image retrieval
• Authors: Chull Hwan Song, Jooyoung Yoon, Shunghyun Choi and Yannis Avrithis
• Abstract summary: Vision transformers have achieved remarkable progress in vision tasks such as image classification and detection. However, in instance-level image retrieval, transformers have not yet shown good performance compared to convolutional networks. We propose a number of improvements that make transformers outperform the state of the art for the first time.
• Score: 11.441395750267052
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision transformers have achieved remarkable progress in vision tasks such as image classification and detection. However, in instance-level image retrieval, transformers have not yet shown good performance compared to convolutional networks. We propose a number of improvements that make transformers outperform the state of the art for the first time. (1) We show that a hybrid architecture is more effective than plain transformers, by a large margin. (2) We introduce two branches collecting global (classification token) and local (patch tokens) information, from which we form a global image representation. (3) In each branch, we collect multi-layer features from the transformer encoder, corresponding to skip connections across distant layers. (4) We enhance locality of interactions at the deeper layers of the encoder, which is the relative weakness of vision transformers. We train our model on all commonly used training sets and, for the first time, we make fair comparisons separately per training set. In all cases, we outperform previous models based on global representation. Public code is available at https://github.com/dealicious-inc/DToP.

Related papers

• Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles [65.54857068975068]
  In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer. We create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models.
  arXiv  Detail & Related papers  (2023-06-01T17:59:58Z)
• On the Surprising Effectiveness of Transformers in Low-Labeled Video Recognition [18.557920268145818]
  Video vision transformers have been shown to be competitive with convolution-based methods (CNNs) broadly across multiple vision tasks. Our work empirically explores the low data regime for video classification and discovers that, surprisingly, transformers perform extremely well in the low-labeled video setting. We even show that using just the labeled data, transformers significantly outperform complex semi-supervised CNN methods that leverage large-scale unlabeled data as well.
  arXiv  Detail & Related papers  (2022-09-15T17:12:30Z)
• TransVG++: End-to-End Visual Grounding with Language Conditioned Vision Transformer [188.00681648113223]
  We explore neat yet effective Transformer-based frameworks for visual grounding. TransVG establishes multi-modal correspondences by Transformers and localizes referred regions by directly regressing box coordinates. We upgrade our framework to a purely Transformer-based one by leveraging Vision Transformer (ViT) for vision feature encoding.
  arXiv  Detail & Related papers  (2022-06-14T06:27:38Z)
• HiViT: Hierarchical Vision Transformer Meets Masked Image Modeling [126.89573619301953]
  We propose a new design of hierarchical vision transformers named HiViT (short for Hierarchical ViT) HiViT enjoys both high efficiency and good performance in MIM. In running MAE on ImageNet-1K, HiViT-B reports a +0.6% accuracy gain over ViT-B and a 1.9$times$ speed-up over Swin-B.
  arXiv  Detail & Related papers  (2022-05-30T09:34:44Z)
• Semi-Supervised Vision Transformers [76.83020291497895]
  We study the training of Vision Transformers for semi-supervised image classification. We find Vision Transformers perform poorly on a semi-supervised ImageNet setting. CNNs achieve superior results in small labeled data regime.
  arXiv  Detail & Related papers  (2021-11-22T09:28:13Z)
• Transformer-Based Deep Image Matching for Generalizable Person Re-identification [114.56752624945142]
  We investigate the possibility of applying Transformers for image matching and metric learning given pairs of images. We find that the Vision Transformer (ViT) and the vanilla Transformer with decoders are not adequate for image matching due to their lack of image-to-image attention. We propose a new simplified decoder, which drops the full attention implementation with the softmax weighting, keeping only the query-key similarity.
  arXiv  Detail & Related papers  (2021-05-30T05:38:33Z)
• Going deeper with Image Transformers [102.61950708108022]
  We build and optimize deeper transformer networks for image classification. We make two transformers architecture changes that significantly improve the accuracy of deep transformers. Our best model establishes the new state of the art on Imagenet with Reassessed labels and Imagenet-V2 / match frequency.
  arXiv  Detail & Related papers  (2021-03-31T17:37:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.