On the Limits of Token Reduction for Efficient Unified Vision Language Training

• URL: http://arxiv.org/abs/2606.01503v1
• Date: Sun, 31 May 2026 23:59:12 GMT
• Title: On the Limits of Token Reduction for Efficient Unified Vision Language Training
• Abstract summary: Unified vision-language models (VLMs) integrate visual understanding and visual generation within a single autoregressive backbone.<n>We study the feasibility and limits of token-reduction-based acceleration for unified VLM training.
• Score: 15.69600993792123
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Unified vision-language models (VLMs) integrate visual understanding and visual generation within a single autoregressive backbone, but their joint training is computationally expensive and largely overlooked from an efficiency perspective. In this work, we study the feasibility and limits of token-reduction-based acceleration for unified VLM training. Through a systematic analysis of layerwise attention allocation, we uncover a fundamental asymmetry: visual understanding exhibits substantial late-layer visual redundancy, whereas visual generation maintains persistent dependence on image tokens across depth. Guided by this observation, we design task-specific accelerators that selectively reduce image-token computation for each objective. While these methods achieve significant efficiency gains in isolated settings, we observe a consistent synergy loss under unified training -- task-specific token dropping necessitates divergent parameter pathways and eliminates the mutual performance gains typically observed in joint optimization. Our findings suggest that efficient unified modeling requires preserving shared cross-task structures, highlighting the need for synergy-aware acceleration strategies. Project page: https://chicychen.github.io/TokenReductionUnifiedVLM/.

Related papers

• Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models [8.482327499828154]
  We propose an attention-free and lightweight token reduction framework as a plug-and-play module for Vision-Language Models.<n>Our framework preserves both important and diverse tokens to produce a compact visual representation.
  arXiv  Detail & Related papers  (2026-07-15T06:55:08Z)
• Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs [89.7106332677868]
  We present Visual-Skip, a training-free inference paradigm that decouples spatial interaction from semantic evolution.<n>V-Skip effectively bypasses redundant vision attention to achieve block-wise sparsity, maintaining a 94.16% to 100.31% performance retention across diverse MLLMs.
  arXiv  Detail & Related papers  (2026-06-07T08:32:13Z)
• VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions [51.41587958253802]
  We introduce VISion On Request (VISOR), a method that reduces inference cost without discarding visual information.<n>VISOR improves efficiency by sparsifying the interaction between image and text tokens.<n>Experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results.
  arXiv  Detail & Related papers  (2026-03-24T17:58:17Z)
• UniCompress: Token Compression for Unified Vision-Language Understanding and Generation [62.943173382496276]
  Unified models aim to support both understanding and generation by encoding images into discrete tokens and processing them alongside text.<n>We propose a unified token compression algorithm that significantly reduces visual token count while preserving performance on both image understanding and generation tasks.
  arXiv  Detail & Related papers  (2026-03-11T21:27:15Z)
• AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models [63.05306474002547]
  Regulatory frameworks mandating the 'right to be forgotten' drive the need for machine unlearning.<n>We introduce AUVIC, a novel visual concept unlearning framework for MLLMs.<n>We show that AUVIC achieves state-of-the-art target forgetting rates while incurs minimal performance degradation on non-target concepts.
  arXiv  Detail & Related papers  (2025-11-14T13:35:32Z)
• Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning [70.57180215148125]
  Visual instruction tuning aims to enable large language models to comprehend the visual world.<n>Existing methods often grapple with the intractable trade-off between accuracy and efficiency.<n>We present LLaVA-Meteor, a novel approach that strategically compresses visual tokens without compromising core information.
  arXiv  Detail & Related papers  (2025-05-17T10:22:29Z)
• End-to-End Vision Tokenizer Tuning [73.3065542220568]
  The vision tokenizer optimized for low-level reconstruction is to downstream tasks requiring varied representations and semantics.<n>The loss of the vision tokenization can be the representation bottleneck for target tasks.<n>We propose ETT, an end-to-end vision tokenizer tuning approach that enables joint optimization between vision tokenization and target autoregressive tasks.
  arXiv  Detail & Related papers  (2025-05-15T17:59:39Z)
• Skip-Vision: Efficient and Scalable Acceleration of Vision-Language Models via Adaptive Token Skipping [13.846838416902575]
  A key bottleneck stems from the proliferation of visual tokens required for fine-grained image understanding.<n>We propose Skip-Vision, a unified framework addressing both training and inference inefficiencies in vision-language models.<n> Experimental results demonstrate that Skip-Vision reduces training time by up to 35%, inference FLOPs by 75%, and latency by 45%.
  arXiv  Detail & Related papers  (2025-03-26T04:16:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.