AdaCodec: A Predictive Visual Code for Video MLLMs

• URL: http://arxiv.org/abs/2606.02569v1
• Date: Mon, 01 Jun 2026 17:56:35 GMT
• Title: AdaCodec: A Predictive Visual Code for Video MLLMs
• Abstract summary: We call this interface a emphpredictive visual code, and instantiate it for video MLLMs as textbfAdaCodec.<n>AdaCodec spends full visual tokens on a reference frame only when its conditional predictive cost is high.<n>It encodes inter-frame changes, including motion and prediction residuals, as compact P-tokens.
• Score: 66.30108093864139
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video is temporally redundant: adjacent frames usually share most objects, background, and layout. Yet existing video multimodal large language models (video MLLMs) usually encode each sampled frame as an independent RGB image, causing visual tokens to repeat content already present in earlier frames. This suggests a more direct video interface: send a full reference frame only when the scene cannot be predicted well from prior context, and otherwise transmit a compact description of inter-frame changes. We call this interface a \emph{predictive visual code}, and instantiate it for video MLLMs as \textbf{AdaCodec}. AdaCodec spends full visual tokens on a reference frame only when its conditional predictive cost is high; otherwise, it encodes inter-frame changes, including motion and prediction residuals, as compact P-tokens. Across all eleven benchmarks, AdaCodec improves over the Qwen3-VL-8B per-frame RGB baseline at a matched visual-token budget. Even at $1/7$ the budget, AdaCodec with 32k tokens surpasses the 224k baseline on all long-video benchmarks; on five general-video benchmarks, it raises the average score while substantially cutting time-to-first-token from 9.26s to 1.62s.

Related papers

• TivTok: Broadcasting Time-Invariant Tokens for Scalable Video Tokenization [33.28254526468271]
  We propose a reuse-aware video tokenizer that makes persistent information reusable across time.<n>TivTok represents a clip with Time-Invariant (TIV) tokens that encode information shared across frames and Time-Variant (TV) tokens that encode frame-specific residuals.
  arXiv  Detail & Related papers  (2026-06-16T06:52:52Z)
• Frames2LoRA: Parametric Video Internalization for Vision-Language Models [52.46510577817688]
  Frames2LoRA is a method for parametric video internalization.<n>A perceiver hypernetwork reads the intermediate representations produced layer-by-layer as a frozen VLM encodes a video.<n>It generates a Low-Rank Adaptation (LoRA) adapter in a single forward pass.
  arXiv  Detail & Related papers  (2026-06-03T02:07:35Z)
• CoPE-VideoLM: Codec Primitives For Efficient Video Language Models [56.76440182038839]
  Video Language Models (VideoLMs) empower AI systems to understand temporal dynamics in videos.<n>Current methods use sampling which can miss both macro-level events and micro-level details due to the sparse temporal coverage.<n>We propose to leverage video primitives which encode video redundancy and sparsity without requiring expensive full-image encoding for most frames.
  arXiv  Detail & Related papers  (2026-02-13T18:57:31Z)
• Seeing the Forest and the Trees: Query-Aware Tokenizer for Long-Video Multimodal Language Models [24.875526594002434]
  We present QTSplus, a visual token selection module for long video understanding scenarios.<n>It is integrated into Qwen2.5-VL and compresses the vision stream by up to textbf89% and reduces end-to-end latency by textbf28% on long videos.<n>Results show that QTSplus is an effective, general mechanism for scaling MLLMs to realworld long-video scenarios.
  arXiv  Detail & Related papers  (2025-11-14T22:41:27Z)
• Multimodal Long Video Modeling Based on Temporal Dynamic Context [13.979661295432964]
  We propose a dynamic long video encoding method utilizing the temporal relationship between frames, named Temporal Dynamic Context (TDC)<n>We segment the video into semantically consistent scenes based on inter-frame similarities, then encode each frame into tokens using visual-audio encoders.<n>To handle extremely long videos, we propose a training-free chain-of-thought strategy that progressively extracts answers from multiple video segments.
  arXiv  Detail & Related papers  (2025-04-14T17:34:06Z)
• Efficient Long Video Tokenization via Coordinate-based Patch Reconstruction [93.69757398746017]
  CoordTok is a video tokenizer that learns a mapping from coordinate-based representations to the corresponding patches of input videos.<n>CoordTok encodes a video into factorized triplane representations and reconstructs patches that correspond to randomly sampled $(x,y,t)$ coordinates.
  arXiv  Detail & Related papers  (2024-11-22T06:50:44Z)
• xGen-MM-Vid (BLIP-3-Video): You Only Need 32 Tokens to Represent a Video Even in VLMs [109.32471628475263]
  We present xGen-MM-Vid (B-3-Video), a multimodal language model for videos.<n>BLIP-3-Video takes advantage of the 'temporal encoder' in addition to the conventional visual tokenizer.<n>We experimentally confirm that BLIP-3-Video obtains video question-answering accuracies comparable to much larger state-of-the-art models.
  arXiv  Detail & Related papers  (2024-10-21T17:59:11Z)
• LLaMA-VID: An Image is Worth 2 Tokens in Large Language Models [66.40252169137447]
  We present a novel method to tackle the token generation challenge in Vision Language Models (VLMs) for video and image understanding, called LLaMA-VID. LLaMA-VID addresses this issue by representing each frame with two distinct tokens, namely context token and content token. This dual-token strategy significantly reduces the overload of long videos while preserving critical information.
  arXiv  Detail & Related papers  (2023-11-28T18:53:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.