TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization

• URL: http://arxiv.org/abs/2606.17590v1
• Date: Tue, 16 Jun 2026 06:52:52 GMT
• Title: TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization
• Abstract summary: We propose a reuse-aware video tokenizer that makes persistent information reusable across time.<n>TivTok represents a clip with Time-Invariant (TIV) tokens that encode information shared across frames and Time-Variant (TV) tokens that encode frame-specific residuals.
• Score: 33.28254526468271
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Video tokenization is fundamental to scalable video generation, as the number of tokens directly determines the computational cost and the length of videos that can be modeled. Existing tokenizers mainly improve scalability by compressing videos into fewer tokens, but they often continue to represent persistent content, such as static backgrounds and consistent object appearances, repeatedly across frames and chunks. In this paper, we propose \textbf{TivTok} (\textit{Time-Invariant Tokenizer}), a reuse-aware video tokenizer that makes persistent information reusable across time. TivTok represents a clip with Time-Invariant (TIV) tokens that encode information shared across frames and Time-Variant (TV) tokens that encode frame-specific residuals. To obtain this factorization, we introduce Scope-Induced Factorization (SIF), which assigns different attention scopes to the two token groups: TIV tokens attend to the full clip, whereas each TV token only accesses its corresponding frame together with the TIV tokens. In the decoder, Invariant Broadcasting (IB) reuses the same TIV tokens across frames and chunks for parallel reconstruction and long-video tokenization. Experiments show that TivTok achieves an rFVD of 12.65 on the standard $16{\times}256{\times}256$ benchmark and improves compression efficiency by 2.91$\times$ for 128-frame videos compared with the evaluated baselines, while using only 1.1\% of the tokens required by downsample-based tokenizers in our evaluation.

Related papers

• AdaCodec: A Predictive Visual Code for Video MLLMs [66.30108093864139]
  We call this interface a emphpredictive visual code, and instantiate it for video MLLMs as textbfAdaCodec.<n>AdaCodec spends full visual tokens on a reference frame only when its conditional predictive cost is high.<n>It encodes inter-frame changes, including motion and prediction residuals, as compact P-tokens.
  arXiv  Detail & Related papers  (2026-06-01T17:56:35Z)
• VTok: A Unified Video Tokenizer with Decoupled Spatial-Temporal Latents [33.80068883432077]
  This work presents VTok, a unified video tokenization framework that can be used for both generation and understanding tasks.<n>We propose to decouple the spatial and temporal representations of videos by retaining the spatial features of a single key frame while encoding each subsequent frame into a single residual token.<n>Our experiments suggest that VTok effectively reduces the complexity of video representation from the product of frame count and per-frame token count to their sum.
  arXiv  Detail & Related papers  (2026-02-04T04:39:46Z)
• Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space [94.07013629356113]
  AdapTok is an adaptive temporal causal video tokenizer that can flexibly allocate tokens for different frames based on video content.<n>AdapTok consistently improves reconstruction quality and generation performance under different token budgets.
  arXiv  Detail & Related papers  (2025-05-22T17:59:02Z)
• VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models [35.38573641029626]
  We introduce the novel task of Extreme Short Token Reduction, which aims to represent entire videos using a minimal set of discrete tokens.<n>On the Extreme Short Token Reduction task, our VQToken compresses sequences to just 0.07 percent of their original length while incurring only a 0.66 percent drop in accuracy on the NextQA-MC benchmark.
  arXiv  Detail & Related papers  (2025-03-21T09:46:31Z)
• Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction [93.69757398746017]
  CoordTok is a video tokenizer that learns a mapping from coordinate-based representations to the corresponding patches of input videos.<n>CoordTok encodes a video into factorized triplane representations and reconstructs patches that correspond to randomly sampled $(x,y,t)$ coordinates.
  arXiv  Detail & Related papers  (2024-11-22T06:50:44Z)
• ElasticTok: Adaptive Tokenization for Image and Video [109.75935878130582]
  We introduce ElasticTok, a method that conditions on prior frames to adaptively encode a frame into a variable number of tokens.<n>During inference, ElasticTok can dynamically allocate tokens when needed.<n>Our evaluations on images and video demonstrate the effectiveness of our approach in efficient token usage.
  arXiv  Detail & Related papers  (2024-10-10T20:54:15Z)
• Efficient Video Action Detection with Token Dropout and Context Refinement [67.10895416008911]
  We propose an end-to-end framework for efficient video action detection (ViTs) In a video clip, we maintain tokens from its perspective while preserving tokens relevant to actor motions from other frames. Second, we refine scene context by leveraging remaining tokens for better recognizing actor identities.
  arXiv  Detail & Related papers  (2023-04-17T17:21:21Z)
• CenterCLIP: Token Clustering for Efficient Text-Video Retrieval [67.21528544724546]
  In CLIP, the essential visual tokenization process, which produces discrete visual token sequences, generates many homogeneous tokens due to the redundancy nature of consecutive frames in videos. This significantly increases computation costs and hinders the deployment of video retrieval models in web applications. In this paper, we design a multi-segment token clustering algorithm to find the most representative tokens and drop the non-essential ones.
  arXiv  Detail & Related papers  (2022-05-02T12:02:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.