Small Vision-Language Models are Smart Compressors for Long Video Understanding

• URL: http://arxiv.org/abs/2604.08120v1
• Date: Thu, 09 Apr 2026 11:40:25 GMT
• Title: Small Vision-Language Models are Smart Compressors for Long Video Understanding
• Abstract summary: Long-form video understanding relies on intent-driven efficiency rather than greedily padded context.<n>We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding.<n>Tests show Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency.
• Score: 73.65465038390771
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the-middle phenomenon. Existing heuristics, like sparse sampling or uniform pooling, blindly sacrifice fidelity by discarding decisive moments and wasting bandwidth on irrelevant backgrounds. We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) as a local temporal compressor, casting token reduction as an early cross-modal distillation process to generate compact, intent-aligned representations in a single forward pass. To enforce strict budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM's zero-shot relevance prior and semantic front-loading, ATA acts as a training-free $O(1)$ dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments show our 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5-16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual budget, outperforming GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames reaches 53.7. Crucially, Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows.

Related papers

• End-to-End Context Compression at Scale [81.70601323130997]
  Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.<n>Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.<n>Existing approaches are not competitive with KV cache compression on the accuracy-efficiency frontier.
  arXiv  Detail & Related papers  (2026-06-08T15:43:16Z)
• One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding [51.08792182064565]
  Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames.<n>With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information.<n>We propose an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling.
  arXiv  Detail & Related papers  (2026-04-15T17:59:52Z)
• PackForcing: Short Video Training Suffices for Long Video Sampling and Long Context Inference [46.18482046594169]
  PackForcing generates coherent 2-minute, 832x480 videos at 16 FPS on a single H200 GPU.<n>It achieves a bounded KV cache of just 4 GB and enables a remarkable 24x temporal (5s to 120s) operating effectively either zero-shot or trained on merely 5-second clips.
  arXiv  Detail & Related papers  (2026-03-26T17:59:05Z)
• Accelerating Streaming Video Large Language Models via Hierarchical Token Compression [12.247532124314402]
  Streaming Video Large Language Models (VideoLLMs) have demonstrated impressive performance across various video understanding tasks.<n>They face significant challenges in real-time deployment due to the high computational cost of processing dense visual tokens from continuous video streams.<n>We propose textbfStreaming textbfToken textbfCompression (textbfSTC), a plug-and-play hierarchical framework that seamlessly integrates into existing streaming VideoLLMs.
  arXiv  Detail & Related papers  (2025-11-30T13:44:28Z)
• StreamingTOM: Streaming Token Compression for Efficient Video Understanding [6.9203477336374775]
  Existing approaches only regulate post-LLM kv-cache, leaving costly pre-LLM prefill unchanged.<n>We introduce StreamingTOM, a training-free, plug-and-play two-stage framework that addresses both pre-LLM and post-LLM bottlenecks with predictable latency.<n> Experiments demonstrate our method achieves $15.7times$ kv-cache compression, $1.2times$ lower peak memory and $2times$ faster TTFT compared to prior SOTA.
  arXiv  Detail & Related papers  (2025-10-21T03:39:41Z)
• Efficient Video Sampling: Pruning Temporally Redundant Tokens for Faster VLM Inference [5.146388234814547]
  Long videos often exceed the token budget of modern language models, leading to severe context limitations and latency issues.<n>We introduce Efficient Video Sampling (EVS), a simple, plug-and-play method for reducing token redundancy in videos by identifying and pruning temporally static patches.<n>EVS substantially reduces token count while maintaining semantic fidelity, enabling faster inference and longer input sequences.
  arXiv  Detail & Related papers  (2025-10-16T12:34:38Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding [55.320254859515714]
  ReTaKe enables VideoLLMs to process 8 times longer frames (up to 2048), similar-sized models by 3-5% and even rivaling much larger ones on VideoMME, MLVU, LongVideoBench, and LVBench.<n>Our code is available at https://github.com/SCZwangxiao/video-ReTaKe.
  arXiv  Detail & Related papers  (2024-12-29T15:42:24Z)
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding [65.46303012350207]
  LongVU is an adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. We leverage DINOv2 features to remove redundant frames that exhibit high similarity. We perform spatial token reduction across frames based on their temporal dependencies.
  arXiv  Detail & Related papers  (2024-10-22T21:21:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.