Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation

• URL: http://arxiv.org/abs/2608.24293v2
• Date: Thu, 27 Aug 2026 05:50:35 GMT
• Title: Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation
• Abstract summary: We present an adaptive token selector that discards uninformative tokens.<n>We empirically show that our model achieves strong reconstruction and generation quality at a state-of-the-art compression ratio.
• Score: 3.7696381512190804
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Latent diffusion models have emerged as a dominant framework for high-fidelity image and video synthesis, operating in compact latent spaces with variational autoencoders (VAEs) to enhance computational efficiency without compromising visual quality. However, conventional VAEs are suboptimal for video data as they employ fixed compression ratios that cannot adapt to the varying complexity of spatio-temporal content. We present KATok (Keep-or-Drop? Adaptive Tokenizer for Compact Video Representation), a transformer-based VAE that incorporates an adaptive token selector which is jointly learned with latent tokens. By evaluating each token's content-richness as keep-or-drop probability, the token selector effectively discards uninformative tokens, naturally allowing data-dependent compression. Applying adaptive tokenization to diffusion models may cause spatial misalignment, as token dropping can disturb the original spatio-temporal structure. To alleviate this issue, we propose two position-prediction strategies: cascaded and joint generation, to ensure spatial consistency. We empirically show that our model achieves strong reconstruction and generation quality at a state-of-the-art compression ratio. Further analysis on video data reveals that this improvement is primarily achieved by reducing spatio-temporal redundancy and removing uninformative tokens, as supported by both quantitative and qualitative results.

Related papers

• CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models [9.660619113725843]
  We propose CRAFT: Compression via Resive Adaptive Fusion Video Tokens.<n>CRAFT consistently outperforms prior state-of-the-art token-compression methods.<n>At about $8times$ compression, it retains roughly $97% of the backbone's average accuracy.
  arXiv  Detail & Related papers  (2026-08-03T03:27:46Z)
• STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation [60.597357847359255]
  Video segmentation demands pixel-accurate object tracking across hundreds of frames under complex natural language queries.<n>Existing methods address this through token compression, yet typically operate on features lacking temporal context.<n>State-space models resolve this constraint, as their linear recurrence selectively conditions each token on temporal context at $mathcalO(T)$ cost compression to reduce.
  arXiv  Detail & Related papers  (2026-07-03T03:28:25Z)
• Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention [23.015486635502437]
  Video language models (Video-LLMs) face high computational costs due to large volumes of visual tokens.<n>We propose a unified selection mechanism that weights semantic similarity to globally select tokens.<n>Unselected tokens are merged via clustering and refilled, preserving information integrity.<n>Our unifiedtemporal token compression strategy establishes the state-of-theart in video understanding under ultra-low token retention.
  arXiv  Detail & Related papers  (2026-03-23T13:15:22Z)
• MTC-VAE: Multi-Level Temporal Compression with Content Awareness [54.85288415164888]
  Latent Video Diffusion Models (LVDMs) rely on Variational Autoencoders (VAEs) to compress videos into compact latent representations.<n>We present a technique to convert fixed compression rate VAEs into models that support multi-level temporal compression.
  arXiv  Detail & Related papers  (2026-02-01T17:08:02Z)
• On the Adversarial Robustness of Large Vision-Language Models under Visual Token Compression [22.436953683970007]
  We show that existing encoder-based attacks can substantially overestimate the robustness of compressed vision-language models (LVLMs)<n>We propose the Compression-AliGnEd attack (CAGE), which aligns perturbation optimization with compression inference without assuming access to the deployed compression mechanism or its token budget.
  arXiv  Detail & Related papers  (2026-01-29T10:47:21Z)
• Adaptive-VoCo: Complexity-Aware Visual Token Compression for Vision-Language Models [19.536595270049016]
  We propose Adaptive-VoCo, a framework that augments VoCo-LLaMA with a lightweight predictor for adaptive compression.<n> Experimental results show that our method consistently outperforms fixed-rate baselines across multiple multimodal tasks.
  arXiv  Detail & Related papers  (2025-12-20T20:24:07Z)
• InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression [114.03378443007074]
  Current tokenizers rigidly compress all content at a fixed rate, leading to redundancy or information loss.<n>This paper introduces InfoTok, a principled framework for adaptive video tokenization.<n>We develop a transformer-based adaptive compressor that enables adaptive tokenization.
  arXiv  Detail & Related papers  (2025-12-18T17:13:59Z)
• Learning Adaptive and Temporally Causal Video Tokenization in a 1D Latent Space [94.07013629356113]
  AdapTok is an adaptive temporal causal video tokenizer that can flexibly allocate tokens for different frames based on video content.<n>AdapTok consistently improves reconstruction quality and generation performance under different token budgets.
  arXiv  Detail & Related papers  (2025-05-22T17:59:02Z)
• REGEN: Learning Compact Video Embedding with (Re-)Generative Decoder [52.698595889988766]
  We present a novel perspective on learning video embedders for generative modeling.<n>Rather than requiring an exact reproduction of an input video, an effective embedder should focus on visually plausible reconstructions.<n>We propose replacing the conventional encoder-decoder video embedder with an encoder-generator framework.
  arXiv  Detail & Related papers  (2025-03-11T17:51:07Z)
• Learned Video Compression via Heterogeneous Deformable Compensation Network [78.72508633457392]
  We propose a learned video compression framework via heterogeneous deformable compensation strategy (HDCVC) to tackle the problems of unstable compression performance. More specifically, the proposed algorithm extracts features from the two adjacent frames to estimate content-Neighborhood heterogeneous deformable (HetDeform) kernel offsets. Experimental results indicate that HDCVC achieves superior performance than the recent state-of-the-art learned video compression approaches.
  arXiv  Detail & Related papers  (2022-07-11T02:31:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.