Video, How Do Your Tokens Merge?

• URL: http://arxiv.org/abs/2506.03885v1
• Date: Wed, 04 Jun 2025 12:28:23 GMT
• Title: Video, How Do Your Tokens Merge?
• Authors: Sam Pollard, Michael Wray,
• Abstract summary: Video transformer models require huge amounts of compute resources due to thetemporal scaling of the input.<n>Recent methods have proposed to drop or merge tokens for image models, whether randomly or via learned methods.<n>Merging tokens has many benefits: it can be plugged into any transformer, vision, and it propagates information that would otherwise be dropped through the model.
• Score: 4.550639073158844
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video transformer models require huge amounts of compute resources due to the spatio-temporal scaling of the input. Tackling this, recent methods have proposed to drop or merge tokens for image models, whether randomly or via learned methods. Merging tokens has many benefits: it can be plugged into any vision transformer, does not require model re-training, and it propagates information that would otherwise be dropped through the model. Before now, video token merging has not been evaluated on temporally complex datasets for video understanding. In this work, we explore training-free token merging for video to provide comprehensive experiments and find best practices across four video transformers on three datasets that exhibit coarse and fine-grained action recognition. Our results showcase the benefits of video token merging with a speedup of around $2.5$X while maintaining accuracy (avg. $-0.55\%$ for ViViT). Code available at https://github.com/sjpollard/video-how-do-your-tokens-merge.

Related papers

• TokensGen: Harnessing Condensed Tokens for Long Video Generation [20.131731700177806]
  TokensGen is a novel framework that leverages condensed tokens to generate long videos.<n>Our method decomposes long video generation into three core tasks: inner-clip semantic control, long-term consistency control, and inter-clip smooth transition.<n> Experimental results demonstrate that our approach significantly enhances long-term temporal and content coherence without incurring prohibitive computational overhead.
  arXiv  Detail & Related papers  (2025-07-21T15:37:33Z)
• Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction [93.69757398746017]
  CoordTok is a video tokenizer that learns a mapping from coordinate-based representations to the corresponding patches of input videos.<n>CoordTok encodes a video into factorized triplane representations and reconstructs patches that correspond to randomly sampled $(x,y,t)$ coordinates.
  arXiv  Detail & Related papers  (2024-11-22T06:50:44Z)
• Principles of Visual Tokens for Efficient Video Understanding [36.05950369461622]
  We propose a lightweight video model, LITE, that can select a small number of tokens effectively.<n>We show that LITE generalizes across datasets and even other tasks without the need for retraining.
  arXiv  Detail & Related papers  (2024-11-20T14:09:47Z)
• Video Token Merging for Long-form Video Understanding [17.59960070514554]
  We propose a learnable video token merging algorithm that dynamically merges tokens based on their saliency. Our approach significantly reduces memory costs by 84% and boosts throughput by approximately 6.89 times compared to baseline algorithms.
  arXiv  Detail & Related papers  (2024-10-31T09:55:32Z)
• ElasticTok: Adaptive Tokenization for Image and Video [109.75935878130582]
  We introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens.<n>During inference, ElasticTok can dynamically allocate tokens when needed.<n>Our evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage.
  arXiv  Detail & Related papers  (2024-10-10T20:54:15Z)
• Token Fusion: Bridging the Gap between Token Pruning and Token Merging [71.84591084401458]
  Vision Transformers (ViTs) have emerged as powerful backbones in computer vision, outperforming many traditional CNNs. computational overhead, largely attributed to the self-attention mechanism, makes deployment on resource-constrained edge devices challenging. We introduce "Token Fusion" (ToFu), a method that amalgamates the benefits of both token pruning and token merging.
  arXiv  Detail & Related papers  (2023-12-02T04:29:19Z)
• UMIFormer: Mining the Correlations between Similar Tokens for Multi-View 3D Reconstruction [9.874357856580447]
  We propose a novel transformer network for Unstructured Multiple Images (UMIFormer) It exploits transformer blocks for decoupled intra-view encoding and designed blocks for token rectification. All tokens acquired from various branches are compressed into a fixed-size compact representation.
  arXiv  Detail & Related papers  (2023-02-27T17:27:45Z)
• VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling [88.30109041658618]
  A great challenge in video-language (VidL) modeling lies in the disconnection between fixed video representations extracted from image/video understanding models and downstream VidL data. We present VIOLET, a fully end-to-end VIdeO-LanguagE Transformer, which adopts a video transformer to explicitly model the temporal dynamics of video inputs.
  arXiv  Detail & Related papers  (2021-11-24T18:31:20Z)
• VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive Learning [82.09856883441044]
  Video understanding relies on perceiving the global content modeling its internal connections. We propose a block-wise strategy where we mask neighboring video tokens in both spatial and temporal domains. We also add an augmentation-free contrastive learning method to further capture global content.
  arXiv  Detail & Related papers  (2021-06-21T16:48:19Z)
• ViViT: A Video Vision Transformer [75.74690759089529]
  We present pure-transformer based models for video classification. Our model extracts-temporal tokens from the input video, which are then encoded by a series of transformer layers. We show how we can effectively regularise the model during training and leverage pretrained image models to be able to train on comparatively small datasets.
  arXiv  Detail & Related papers  (2021-03-29T15:27:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.