One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding

• URL: http://arxiv.org/abs/2604.14149v2
• Date: Thu, 16 Apr 2026 15:48:38 GMT
• Title: One Token per Highly Selective Frame: Towards Extreme Compression for Long Video Understanding
• Abstract summary: Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames.<n>With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information.<n>We propose an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling.
• Score: 51.08792182064565
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long video understanding is inherently challenging for vision-language models (VLMs) because of the extensive number of frames. With each video frame typically expanding into tens or hundreds of tokens, the limited context length of large language models (LLMs) forces the VLMs to perceive the frames sparsely and lose temporal information. To address this, we explore extreme video token compression towards one token per frame at the final LLM layer. Our key insight is that heuristic-based compression, widely adopted by previous methods, is prone to information loss, and this necessitates supervising LLM layers into learnable and progressive modules for token-level compression (LP-Comp). Such compression enables our VLM to digest 2x-4x more frames with improved performance. To further increase the token efficiency, we investigate frame-level compression, which selects the frames most relevant to the queries via the internal attention scores of the LLM layers, named question-conditioned compression (QC-Comp). As a notable distinction from previous studies, we mitigate the position bias of LLM attention in long contexts, i.e., the over-concentration on the beginning and end of a sequence, by splitting long videos into short segments and employing local attention. Collectively, our combined token-level and frame-level leads to an extreme compression model for long video understanding, named XComp, achieving a significantly larger compression ratio and enabling denser frame sampling. Our XComp is finetuned from VideoChat-Flash with a data-efficient supervised compression tuning stage that only requires 2.5% of the supervised fine-tuning data, yet boosts the accuracy from 42.9% to 46.2% on LVBench and enhances multiple other long video benchmarks.

Related papers

• End-to-End Context Compression at Scale [81.70601323130997]
  Long-context language model inference is bottlenecked by memory, as the KV cache grows with context length.<n>Recent techniques to compress the KV cache fall short: they either degrade model quality substantially or require considerable time and compute to compress a single long prompt.<n>Existing approaches are not competitive with KV cache compression on the accuracy-efficiency frontier.
  arXiv  Detail & Related papers  (2026-06-08T15:43:16Z)
• LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs [90.77662862634509]
  LiteFrame is a strong, yet highly efficient backbone for Video Large Language Models.<n>LiteFrame provides a 35% reduction in end-to-end latency while processing 8$times$ more frames.<n>Our results demonstrate a potential path to unlocking longer-form video understanding under fixed compute budgets.
  arXiv  Detail & Related papers  (2026-05-17T05:02:52Z)
• CoPE-VideoLM: Codec Primitives For Efficient Video Language Models [56.76440182038839]
  Video Language Models (VideoLMs) empower AI systems to understand temporal dynamics in videos.<n>Current methods use sampling which can miss both macro-level events and micro-level details due to the sparse temporal coverage.<n>We propose to leverage video primitives which encode video redundancy and sparsity without requiring expensive full-image encoding for most frames.
  arXiv  Detail & Related papers  (2026-02-13T18:57:31Z)
• Less Is More, but Where? Dynamic Token Compression via LLM-Guided Keyframe Prior [31.997025910713077]
  We propose Dynamic Token compression via LLM-guided Keyframe prior (DyToK)<n>Our analysis reveals that VLLM attention layers naturally encoding query-conditioned priors, by which DyToK dynamically adjusts per-frame token retention ratios.<n>Experiments demonstrate that DyToK achieves state-of-the-art efficiency-accuracy tradeoffs.
  arXiv  Detail & Related papers  (2025-12-07T14:42:10Z)
• Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models [24.875526594002434]
  We present QTSplus, a visual token selection module for long video understanding scenarios.<n>It is integrated into Qwen2.5-VL and compresses the vision stream by up to textbf89% and reduces end-to-end latency by textbf28% on long videos.<n>Results show that QTSplus is an effective, general mechanism for scaling MLLMs to realworld long-video scenarios.
  arXiv  Detail & Related papers  (2025-11-14T22:41:27Z)
• MARC: Memory-Augmented RL Token Compression for Efficient Video Understanding [13.02027465520324]
  We propose MARC, which integrates structured retrieval and RL-based distillation.<n>MARC achieves near-baseline accuracy using only one frame's tokens.<n>This demonstrates its potential for efficient, real-time video understanding in resource-constrained settings.
  arXiv  Detail & Related papers  (2025-10-09T08:07:19Z)
• LVC: A Lightweight Compression Framework for Enhancing VLMs in Long Video Understanding [29.719450799231705]
  Vision-Language Models (VLMs) obtain frame-level understanding capabilities through multi-frame input.<n>Video Large Language Models (Video-LLMs) capture temporal relationships within visual features but are limited by the scarcity of high-quality video-text datasets.<n>We propose Lightweight Video Compression (LVC), a novel method featuring the Query-Attention Video Compression mechanism.
  arXiv  Detail & Related papers  (2025-04-09T12:51:10Z)
• STORM: Token-Efficient Long Video Understanding for Multimodal LLMs [116.4479155699528]
  STORM is a novel architecture incorporating a dedicated temporal encoder between the image encoder and the Video-LLMs.<n>We show that STORM achieves state-of-the-art results across various long video understanding benchmarks.
  arXiv  Detail & Related papers  (2025-03-06T06:17:38Z)
• ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding [55.320254859515714]
  ReTaKe enables VideoLLMs to process 8 times longer frames (up to 2048), similar-sized models by 3-5% and even rivaling much larger ones on VideoMME, MLVU, LongVideoBench, and LVBench.<n>Our code is available at https://github.com/SCZwangxiao/video-ReTaKe.
  arXiv  Detail & Related papers  (2024-12-29T15:42:24Z)
• LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding [65.46303012350207]
  LongVU is an adaptive compression mechanism that reduces the number of video tokens while preserving visual details of long videos. We leverage DINOv2 features to remove redundant frames that exhibit high similarity. We perform spatial token reduction across frames based on their temporal dependencies.
  arXiv  Detail & Related papers  (2024-10-22T21:21:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.