Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention

• URL: http://arxiv.org/abs/2603.21957v1
• Date: Mon, 23 Mar 2026 13:15:22 GMT
• Title: Unified Spatiotemporal Token Compression for Video-LLMs at Ultra-Low Retention
• Abstract summary: Video language models (Video-LLMs) face high computational costs due to large volumes of visual tokens.<n>We propose a unified selection mechanism that weights semantic similarity to globally select tokens.<n>Unselected tokens are merged via clustering and refilled, preserving information integrity.<n>Our unifiedtemporal token compression strategy establishes the state-of-theart in video understanding under ultra-low token retention.
• Score: 23.015486635502437
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video large language models (Video-LLMs) face high computational costs due to large volumes of visual tokens. Existing token compression methods typically adopt a two-stage spatiotemporal compression strategy, relying on stage-specific metrics and an implicit assumption of spatiotemporal separability. Under extremely low retention ratios, however, such approaches often result in unbalanced allocation and loss of visual evidence essential for question answering. We reformulate token compression as a spatiotemporal allocation task within a global token retention pool. We propose a unified selection mechanism that integrates attention weights and semantic similarity to globally select tokens with high contribution and low redundancy. Unselected tokens are merged via clustering and refilled, preserving information integrity. Inside the LLM, we further introduce text-aware merging to perform secondary compression based on query relevance. Without requiring retraining, our method serves as a plug-and-play module compatible with existing Video-LLMs. Experiments show that retaining only about 2% of visual tokens preserves 90.1% of baseline performance across multiple benchmarks, while reducing FLOPs to roughly 2.6%. These benefits generalize across diverse backbones, decreasing end-to-end inference latency and memory consumption. Our unified spatiotemporal token compression strategy establishes the state-of-the-art in video understanding under ultra-low token retention.

Related papers

• Rethinking Video Token Compression with a Global Codebook: Learning Once, Compressing Everywhere [5.7189025167000205]
  Video large language models (Video-LLMs) represent videos as dense sequences of visual tokens, whose length grows with the temporal and spatial extent of the input.<n>Existing token compression methods, including pruning and merging, perform compression online during inference.<n>We propose textbfONCE, a plug-in video token compression framework that introduces an offline-to-online paradigm.
  arXiv  Detail & Related papers  (2026-08-02T14:24:13Z)
• Think in Sets for Streaming Video Token Compression [11.833562937219753]
  Streaming video makes causal, frame-by-frame pruning decisions, so modeling cross-frame interactions requires an explicit historical reference.<n>We introduce NovaCov, to our knowledge the first training-free, plug-and-play set-wise token compressor designed for streaming video.
  arXiv  Detail & Related papers  (2026-08-02T11:44:51Z)
• VisCo: Leveraging Large Language Models as Intrinsic Encoders for Visual Token Compression [52.23010472235226]
  Vision-language models (VLMs) process large numbers of visual tokens, resulting in substantial inference and overhead latency.<n>We propose VisCo, a training-efficient self-compression framework that reuses the pretrained VLM itself as an intrinsic compressor.
  arXiv  Detail & Related papers  (2026-07-14T13:21:36Z)
• STAC: Selective Spatiotemporal Aggregation and Compression for Video Reasoning Segmentation [60.597357847359255]
  Video segmentation demands pixel-accurate object tracking across hundreds of frames under complex natural language queries.<n>Existing methods address this through token compression, yet typically operate on features lacking temporal context.<n>State-space models resolve this constraint, as their linear recurrence selectively conditions each token on temporal context at $mathcalO(T)$ cost compression to reduce.
  arXiv  Detail & Related papers  (2026-07-03T03:28:25Z)
• One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding [51.08792182064565]
  Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames.<n>With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information.<n>We propose an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling.
  arXiv  Detail & Related papers  (2026-04-15T17:59:52Z)
• KiToke: Kernel-based Interval-aware Token Compression for Video Large Language Models [12.489375650998554]
  KiToke is a training-free, query-agnostic token compression approach.<n>Our method estimates token diversity globally using a kernel-based measure.<n>Our experiments show that KiToke consistently outperforms existing training-free compression methods.
  arXiv  Detail & Related papers  (2026-04-03T19:18:22Z)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity [54.95089105944234]
  Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens.<n>We propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline.<n>We demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT.
  arXiv  Detail & Related papers  (2026-03-10T10:31:58Z)
• VisionSelector: End-to-End Learnable Visual Token Compression for Efficient Multimodal LLMs [82.72388893596555]
  Multimodal Large Language Models (MLLMs) encounter significant computational and memory bottlenecks.<n>Previous token compression techniques are often constrained by rules that risk discarding critical information.<n>We reformulate token compression as a lightweight plug-and-play framework that reformulates token compression into an end-to-end learnable decision process.
  arXiv  Detail & Related papers  (2025-10-18T17:54:18Z)
• QuantSparse: Comprehensively Compressing Video Diffusion Transformer with Model Quantization and Attention Sparsification [67.15451442018258]
  Diffusion transformers exhibit remarkable video generation capability, yet their prohibitive computational and memory costs hinder practical deployment.<n>Model quantization and attention sparsification are two promising directions for compression, but each alone suffers severe performance degradation under aggressive compression.<n>We propose textbfQuantSparse, a unified framework that integrates model quantization with attention sparsification.
  arXiv  Detail & Related papers  (2025-09-28T06:49:44Z)
• VQToken: Neural Discrete Token Representation Learning for Extreme Token Reduction in Video Large Language Models [35.38573641029626]
  We introduce the novel task of Extreme Short Token Reduction, which aims to represent entire videos using a minimal set of discrete tokens.<n>On the Extreme Short Token Reduction task, our VQToken compresses sequences to just 0.07 percent of their original length while incurring only a 0.66 percent drop in accuracy on the NextQA-MC benchmark.
  arXiv  Detail & Related papers  (2025-03-21T09:46:31Z)
• Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models [36.16630765077807]
  We propose a Hybrid-level Instruction Injection Strategy for Conditional Token Compression in MLLMs (HICom)<n>We use the instruction as a condition to guide the compression from both local and global levels.<n>Experiments show that our HICom can obtain distinguished video understanding ability with fewer tokens.
  arXiv  Detail & Related papers  (2025-03-20T11:09:18Z)
• Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Models [21.36437021964681]
  "Global Compression Commander" is a novel token compression framework for HR-LVLMs.<n>GlobalCom$2$ maintains over 90% performance while compressing 90% visual tokens, reducing FLOPs and peak memory to 9.1% and 60%.
  arXiv  Detail & Related papers  (2025-01-09T11:57:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.