Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models

• URL: http://arxiv.org/abs/2608.28138v1
• Date: Fri, 28 Aug 2026 09:59:27 GMT
• Title: Token-Budget Distillation: Transferring Full-Token Semantics to Compressed Video Vision-Language Models
• Abstract summary: Token-Budget Distillation (TBD) is a parameter-efficient fine-tuning framework for adapting video vision-language models.<n>We evaluate TBD on three video VLM backbones, including LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct.
• Score: 20.77624638873576
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adapting video vision-language models (VLMs) is computationally expensive because video inputs produce a large number of visual tokens, making both fine-tuning and inference costly. Although visual token compression can reduce this overhead, direct adaptation on compressed inputs often causes semantic drift and noticeable performance degradation. We present Token-Budget Distillation (TBD), a parameter-efficient fine-tuning framework for adapting video VLMs under a fixed token budget. TBD freezes the pretrained backbone, updates only LoRA adapters, and integrates FlashVID-based visual token compression into the video pathway. To preserve full-token semantics under compression, TBD employs a dual-path teacher-student design, where a full-token teacher provides stable supervision and a compressed student is optimized with task loss, answer-region KL distillation, GT-anchored margin distillation, and reliability-aware KD control. This design enables the student to recover the semantic behavior of the full-token model while remaining efficient under aggressive token reduction. We evaluate TBD on three video VLM backbones, including LLaVA-Video, LLaVA-OneVision, and Qwen3-VL-8B-Instruct, across four video understanding benchmarks. TBD consistently outperforms compression-only baselines under both moderate and aggressive compression. On LLaVA-Video at retention ratio R = 10 percent, TBD preserves 97.0 percent of the Vanilla model's average accuracy; on LLaVA-OneVision at R = 10 percent, it achieves an average score of 58.4 and matches 100.0 percent relative accuracy.

Related papers

• CRAFT: Compression via Recursive Adaptive Fusion of Video Tokens for Vision-Language Models [9.660619113725843]
  We propose CRAFT: Compression via Resive Adaptive Fusion Video Tokens.<n>CRAFT consistently outperforms prior state-of-the-art token-compression methods.<n>At about $8times$ compression, it retains roughly $97% of the backbone's average accuracy.
  arXiv  Detail & Related papers  (2026-08-03T03:27:46Z)
• LiteFrame: Efficient Vision Encoders Unlock Frame Scaling in Video LLMs [90.77662862634509]
  LiteFrame is a strong, yet highly efficient backbone for Video Large Language Models.<n>LiteFrame provides a 35% reduction in end-to-end latency while processing 8$times$ more frames.<n>Our results demonstrate a potential path to unlocking longer-form video understanding under fixed compute budgets.
  arXiv  Detail & Related papers  (2026-05-17T05:02:52Z)
• Small Vision-Language Models are Smart Compressors for Long Video Understanding [73.65465038390771]
  Long-form video understanding relies on intent-driven efficiency rather than greedily padded context.<n>We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding.<n>Tests show Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency.
  arXiv  Detail & Related papers  (2026-04-09T11:40:25Z)
• DUET-VLM: Dual stage Unified Efficient Token reduction for VLM Training and Inference [14.714791872881397]
  DUET-VLM is a versatile plug-and-play dual compression framework.<n>It enables robust adaptation to reduced visual (image/video) input without sacrificing accuracy.<n>Our results highlight end-to-end training with DUET-VLM.
  arXiv  Detail & Related papers  (2026-02-21T14:22:49Z)
• PIO-FVLM: Rethinking Training-Free Visual Token Reduction for VLM Acceleration from an Inference-Objective Perspective [59.24570811503256]
  We propose PIO-FVLM to reduce redundant visual tokens in vision-models (VLMs) to accelerate inference.<n>The proposed PIO-FVLM is training-free and compatible with FlashAttention, friendly to practical application and deployment.<n>On LLaVA-Next-7B, PIO-FVLM retains just 11.1% of visual tokens but maintains 97.2% of the original performance.
  arXiv  Detail & Related papers  (2026-02-04T15:33:10Z)
• MARC: Memory-Augmented RL Token Compression for Efficient Video Understanding [13.02027465520324]
  We propose MARC, which integrates structured retrieval and RL-based distillation.<n>MARC achieves near-baseline accuracy using only one frame's tokens.<n>This demonstrates its potential for efficient, real-time video understanding in resource-constrained settings.
  arXiv  Detail & Related papers  (2025-10-09T08:07:19Z)
• METok: Multi-Stage Event-based Token Compression for Efficient Long Video Understanding [55.38256656122857]
  We propose METok, a training-free, Multi-stage Event-based Token compression framework.<n>We show METok achieves an optimal trade-off between efficiency and accuracy by dynamically selecting informative visual tokens.<n>For instance, equipping LongVA-7B with METok realizes an 80.6% FLOPs reduction and 93.5% KV Cache memory savings.
  arXiv  Detail & Related papers  (2025-06-03T13:19:41Z)
• ReTaKe: Reducing Temporal and Knowledge Redundancy for Long Video Understanding [55.320254859515714]
  ReTaKe enables VideoLLMs to process 8 times longer frames (up to 2048), similar-sized models by 3-5% and even rivaling much larger ones on VideoMME, MLVU, LongVideoBench, and LVBench.<n>Our code is available at https://github.com/SCZwangxiao/video-ReTaKe.
  arXiv  Detail & Related papers  (2024-12-29T15:42:24Z)
• Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning [63.43972993473501]
  Token compression expedites the training and inference of Vision Transformers (ViTs) However, when applied to downstream tasks, compression degrees are mismatched between training and inference stages. We propose a model arithmetic framework to decouple the compression degrees between the two stages.
  arXiv  Detail & Related papers  (2024-08-13T10:36:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.