Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer

• URL: http://arxiv.org/abs/2408.17062v1
• Date: Fri, 30 Aug 2024 07:48:05 GMT
• Title: Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer
• Abstract summary: Vote&Mix (textbfVoMix), a plug-and-play and parameter-free token reduction method, can be readily applied to off-the-shelf ViT models. VoMix tackles the computational redundancy of ViTs by identifying tokens with high homogeneity through a layer-wise token similarity voting mechanism. Experiments demonstrate VoMix significantly improves the speed-accuracy tradeoff of ViTs on both images and videos.
• Score: 17.724361544581527
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Despite the remarkable success of Vision Transformers (ViTs) in various visual tasks, they are often hindered by substantial computational cost. In this work, we introduce Vote\&Mix (\textbf{VoMix}), a plug-and-play and parameter-free token reduction method, which can be readily applied to off-the-shelf ViT models \textit{without any training}. VoMix tackles the computational redundancy of ViTs by identifying tokens with high homogeneity through a layer-wise token similarity voting mechanism. Subsequently, the selected tokens are mixed into the retained set, thereby preserving visual information. Experiments demonstrate VoMix significantly improves the speed-accuracy tradeoff of ViTs on both images and videos. Without any training, VoMix achieves a 2$\times$ increase in throughput of existing ViT-H on ImageNet-1K and a 2.4$\times$ increase in throughput of existing ViT-L on Kinetics-400 video dataset, with a mere 0.3\% drop in top-1 accuracy.

Related papers

• LookupViT: Compressing visual information to a limited number of tokens [36.83826969693139]
  Vision Transformers (ViT) have emerged as the de-facto choice for numerous industry grade vision solutions. But their inference cost can be prohibitive for many settings, as they compute self-attention in each layer which suffers from complexity in the number of tokens. In this work, we introduce LookupViT, that exploits this information sparsity to reduce ViT inference cost.
  arXiv  Detail & Related papers  (2024-07-17T17:22:43Z)
• SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection [3.960622297616708]
  We propose a method to optimize the amount of unnecessary interactions between unimportant tokens by separating and sending them through a different low-cost computational path. Our experimental results on training ViT-small from scratch show that SkipViT is capable of effectively dropping 55% of the tokens while gaining more than 13% training throughput.
  arXiv  Detail & Related papers  (2024-01-27T04:24:49Z)
• No Token Left Behind: Efficient Vision Transformer via Dynamic Token Idling [55.203866875294516]
  Vision Transformers (ViTs) have demonstrated outstanding performance in computer vision tasks. Various token pruning techniques have been introduced to alleviate the high computational burden of ViTs. We propose IdleViT, a dynamic token-idle-based method that achieves an excellent trade-off between performance and efficiency.
  arXiv  Detail & Related papers  (2023-10-09T12:10:41Z)
• CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs [79.54107547233625]
  Vision Transformers (ViTs) have emerged as state-of-the-art models for various vision tasks. We propose a joint compression method for ViTs that offers both high accuracy and fast inference speed. Our proposed method can achieve state-of-the-art performance across various ViTs.
  arXiv  Detail & Related papers  (2023-09-27T16:12:07Z)
• Dynamic Token-Pass Transformers for Semantic Segmentation [22.673910995773262]
  We introduce dynamic token-pass vision transformers (DoViT) for semantic segmentation. DoViT gradually stops partial easy tokens from self-attention calculation and keeps the hard tokens forwarding until meeting the stopping criteria. Our method greatly reduces about 40% $sim$ 60% FLOPs and the drop of mIoU is within 0.8% for various segmentation transformers.
  arXiv  Detail & Related papers  (2023-08-03T06:14:24Z)
• SMMix: Self-Motivated Image Mixing for Vision Transformers [65.809376136455]
  CutMix is a vital augmentation strategy that determines the performance and generalization ability of vision transformers (ViTs) Existing CutMix variants tackle this problem by generating more consistent mixed images or more precise mixed labels. We propose an efficient and effective Self-Motivated image Mixing method (SMMix) which motivates both image and label enhancement by the model under training itself.
  arXiv  Detail & Related papers  (2022-12-26T00:19:39Z)
• Token Merging: Your ViT But Faster [33.61038274945422]
  ToMe gradually combines similar tokens in a transformer using a general and light-weight matching algorithm. Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images. Training with ToMe further minimizes accuracy drop, leading to 2x the throughput of ViT-B on audio for only a 0.4% mAP drop.
  arXiv  Detail & Related papers  (2022-10-17T22:23:40Z)
• TokenMix: Rethinking Image Mixing for Data Augmentation in Vision Transformers [36.630476419392046]
  CutMix is a popular augmentation technique commonly used for training modern convolutional and transformer vision networks. We propose a novel data augmentation technique TokenMix to improve the performance of vision transformers.
  arXiv  Detail & Related papers  (2022-07-18T07:08:29Z)
• LightViT: Towards Light-Weight Convolution-Free Vision Transformers [43.48734363817069]
  Vision transformers (ViTs) are usually considered to be less light-weight than convolutional neural networks (CNNs) We present LightViT as a new family of light-weight ViTs to achieve better accuracy-efficiency balance upon the pure transformer blocks without convolution. Experiments show that our model achieves significant improvements on image classification, object detection, and semantic segmentation tasks.
  arXiv  Detail & Related papers  (2022-07-12T14:27:57Z)
• Super Vision Transformer [131.4777773281238]
  Experimental results on ImageNet demonstrate that our SuperViT can considerably reduce the computational costs of ViT models with even performance increase. Our SuperViT significantly outperforms existing studies on efficient vision transformers.
  arXiv  Detail & Related papers  (2022-05-23T15:42:12Z)
• Auto-scaling Vision Transformers without Training [84.34662535276898]
  We propose As-ViT, an auto-scaling framework for Vision Transformers (ViTs) without training. As-ViT automatically discovers and scales up ViTs in an efficient and principled manner. As a unified framework, As-ViT achieves strong performance on classification and detection.
  arXiv  Detail & Related papers  (2022-02-24T06:30:55Z)
• Patches Are All You Need? [96.88889685873106]
  Vision Transformer (ViT) models may exceed their performance in some settings. ViTs require the use of patch embeddings, which group together small regions of the image into single input features. This raises a question: Is the performance of ViTs due to the inherently-more-powerful Transformer architecture, or is it at least partly due to using patches as the input representation?
  arXiv  Detail & Related papers  (2022-01-24T16:42:56Z)
• Self-slimmed Vision Transformer [52.67243496139175]
  Vision transformers (ViTs) have become the popular structures and outperformed convolutional neural networks (CNNs) on various vision tasks. We propose a generic self-slimmed learning approach for vanilla ViTs, namely SiT. Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs.
  arXiv  Detail & Related papers  (2021-11-24T16:48:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.