Token Merging: Your ViT But Faster

• URL: http://arxiv.org/abs/2210.09461v1
• Date: Mon, 17 Oct 2022 22:23:40 GMT
• Title: Token Merging: Your ViT But Faster
• Abstract summary: ToMe gradually combines similar tokens in a transformer using a general and light-weight matching algorithm. Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images. Training with ToMe further minimizes accuracy drop, leading to 2x the throughput of ViT-B on audio for only a 0.4% mAP drop.
• Score: 33.61038274945422
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Token Merging (ToMe), a simple method to increase the throughput of existing ViT models without needing to train. ToMe gradually combines similar tokens in a transformer using a general and light-weight matching algorithm that is as fast as pruning while being more accurate. Off-the-shelf, ToMe can 2x the throughput of state-of-the-art ViT-L @ 512 and ViT-H @ 518 models on images and 2.2x the throughput of ViT-L on video with only a 0.2-0.3% accuracy drop in each case. ToMe can also easily be applied during training, improving in practice training speed up to 2x for MAE fine-tuning on video. Training with ToMe further minimizes accuracy drop, leading to 2x the throughput of ViT-B on audio for only a 0.4% mAP drop. Qualitatively, we find that ToMe merges object parts into one token, even over multiple frames of video. Overall, ToMe's accuracy and speed are competitive with state-of-the-art on images, video, and audio.

Related papers

• MaMe & MaRe: Matrix-Based Token Merging and Restoration for Efficient Visual Perception and Synthesis [2.5885108031811006]
  Token compression is crucial for mitigating the quadratic complexity of self-attention mechanisms in Vision Transformers (ViTs)<n>We introduce MaMe, a training-free, differentiable token merging method based entirely on matrix operations.<n>We present MaRe, its inverse operation, for token restoration, forming a MaMe+MaRe pipeline for image synthesis.
  arXiv  Detail & Related papers  (2026-04-15T03:06:24Z)
• Vote&Mix: Plug-and-Play Token Reduction for Efficient Vision Transformer [17.724361544581527]
  Vote&Mix (textbfVoMix), a plug-and-play and parameter-free token reduction method, can be readily applied to off-the-shelf ViT models. VoMix tackles the computational redundancy of ViTs by identifying tokens with high homogeneity through a layer-wise token similarity voting mechanism. Experiments demonstrate VoMix significantly improves the speed-accuracy tradeoff of ViTs on both images and videos.
  arXiv  Detail & Related papers  (2024-08-30T07:48:05Z)
• SkipViT: Speeding Up Vision Transformers with a Token-Level Skip Connection [3.960622297616708]
  We propose a method to optimize the amount of unnecessary interactions between unimportant tokens by separating and sending them through a different low-cost computational path. Our experimental results on training ViT-small from scratch show that SkipViT is capable of effectively dropping 55% of the tokens while gaining more than 13% training throughput.
  arXiv  Detail & Related papers  (2024-01-27T04:24:49Z)
• CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs [79.54107547233625]
  Vision Transformers (ViTs) have emerged as state-of-the-art models for various vision tasks. We propose a joint compression method for ViTs that offers both high accuracy and fast inference speed. Our proposed method can achieve state-of-the-art performance across various ViTs.
  arXiv  Detail & Related papers  (2023-09-27T16:12:07Z)
• Masked Image Residual Learning for Scaling Deeper Vision Transformers [37.313915004909184]
  We introduce a self-supervised learning framework called Masked Image Residual Learning (MIRL) to ease the training of deeper ViTs. MIRL significantly alleviates the degradation problem, making scaling ViT along depth a promising direction for performance upgrade. With less pre-training time, MIRL yields competitive performance compared to other approaches.
  arXiv  Detail & Related papers  (2023-09-25T13:45:28Z)
• UniFormerV2: Spatiotemporal Learning by Arming Image ViTs with Video UniFormer [63.53290944664774]
  Vision Transformers (ViTs) have shown their power in learning long-term video dependency with self-attention. UniFormer has successfully alleviated this issue, by unifying convolution and self-attention as a relation aggregator in the transformer format. We propose a generic paradigm to build a powerful family of video networks, by arming the pretrainedTs with efficient UniFormer designs.
  arXiv  Detail & Related papers  (2022-11-17T14:17:40Z)
• Super Vision Transformer [131.4777773281238]
  Experimental results on ImageNet demonstrate that our SuperViT can considerably reduce the computational costs of ViT models with even performance increase. Our SuperViT significantly outperforms existing studies on efficient vision transformers.
  arXiv  Detail & Related papers  (2022-05-23T15:42:12Z)
• Auto-scaling Vision Transformers without Training [84.34662535276898]
  We propose As-ViT, an auto-scaling framework for Vision Transformers (ViTs) without training. As-ViT automatically discovers and scales up ViTs in an efficient and principled manner. As a unified framework, As-ViT achieves strong performance on classification and detection.
  arXiv  Detail & Related papers  (2022-02-24T06:30:55Z)
• Self-slimmed Vision Transformer [52.67243496139175]
  Vision transformers (ViTs) have become the popular structures and outperformed convolutional neural networks (CNNs) on various vision tasks. We propose a generic self-slimmed learning approach for vanilla ViTs, namely SiT. Specifically, we first design a novel Token Slimming Module (TSM), which can boost the inference efficiency of ViTs.
  arXiv  Detail & Related papers  (2021-11-24T16:48:57Z)
• Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNet [128.96032932640364]
  We propose a new Tokens-To-Token Vision Transformers (T2T-ViT) to solve vision tasks. T2T-ViT reduces the parameter counts and MACs of vanilla ViT by 200%, while achieving more than 2.5% improvement when trained from scratch on ImageNet. For example, T2T-ViT with ResNet50 comparable size can achieve 80.7% top-1 accuracy on ImageNet.
  arXiv  Detail & Related papers  (2021-01-28T13:25:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.