Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models

• URL: http://arxiv.org/abs/2607.13500v1
• Date: Wed, 15 Jul 2026 06:55:08 GMT
• Title: Attention-Free and Lightweight Token Reduction for Efficient Vision-Language Models
• Abstract summary: We propose an attention-free and lightweight token reduction framework as a plug-and-play module for Vision-Language Models.<n>Our framework preserves both important and diverse tokens to produce a compact visual representation.
• Score: 8.482327499828154
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) have achieved strong performance in multimodal understanding, yet remain challenging to deploy on resource-constrained edge devices due to the substantial computational overhead of processing numerous visual tokens. Token reduction is a promising direction for accelerating VLMs inference, but existing approaches either rely on attention maps that are incompatible with modern acceleration frameworks or depend on computationally intensive pairwise similarity comparisons, which undermine scalability and negate their practical benefits in deployment. In this paper, we propose an attention-free and lightweight token reduction framework as a plug-and-play module for VLMs, which preserves both important and diverse tokens to produce a compact visual representation. First, to enable attention-free importance estimation, we adopt an information-theoretic perspective and quantify token information using a novel entropy-based criterion, retaining those with more expressive and less degenerate feature representations. Second, to ensure diverse visual coverage in a lightweight manner, we introduce a transformation-induced consistency signal where similar tokens yield similar signals, such that sorting by this signal places similar tokens close to each other and enables stride-based selection to produce a diverse token set. Extensive experiments across multiple VLMs benchmarks demonstrate that our framework achieves a favorable accuracy-efficiency trade-off, maintaining competitive performance under aggressive compression.

Related papers

• TORINO: Token Reduction via Interpretable Concept Overlap in Vision-Language Models [6.132992639292891]
  Vision-Language Models (VLMs) have demonstrated impressive capabilities across different tasks, but their computational cost is dominated by the large number of visual tokens fed to the model.<n>We introduce TORINO, a plug-and-play framework for adaptive visual token reduction in VLMs that requires no fine-tuning of the underlying model.
  arXiv  Detail & Related papers  (2026-07-06T01:43:09Z)
• Bridging Perception and Reasoning: Token Reweighting for RLVR in Multimodal LLMs [51.60575965819268]
  We propose a plug-and-play Token-Reweighting (ToR) strategy that explicitly models this interdependence.<n>ToR delivers consistent performance gains across multiple multi-modal reasoning benchmarks.
  arXiv  Detail & Related papers  (2026-03-26T06:25:27Z)
• VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions [51.41587958253802]
  We introduce VISion On Request (VISOR), a method that reduces inference cost without discarding visual information.<n>VISOR improves efficiency by sparsifying the interaction between image and text tokens.<n>Experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results.
  arXiv  Detail & Related papers  (2026-03-24T17:58:17Z)
• UniCompress: Token Compression for Unified Vision-Language Understanding and Generation [62.943173382496276]
  Unified models aim to support both understanding and generation by encoding images into discrete tokens and processing them alongside text.<n>We propose a unified token compression algorithm that significantly reduces visual token count while preserving performance on both image understanding and generation tasks.
  arXiv  Detail & Related papers  (2026-03-11T21:27:15Z)
• ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning [8.933549837045932]
  Large Vision-Language Models incur high computational costs due to significant redundancy in their visual tokens.<n>We propose a Visual and Textual Semantic Collaborative Pruning framework (ViTCoP) that combines redundancy filtering in the vision encoder with step-wise co-pruning within the Large Language Models.
  arXiv  Detail & Related papers  (2026-01-25T12:47:30Z)
• SPOT: Sparsification with Attention Dynamics via Token Relevance in Vision Transformers [9.24617274894642]
  We present SParsification with attentiOn dynamics via Token relevance (SPOT) as a framework for early detection of redundant tokens within Vision Transformers (ViTs)<n>SPOT informs token sparsification and facilitates the elimination of such tokens, improving computational efficiency without sacrificing performance.<n> Empirical evaluations demonstrate significant efficiency gains of up to 40% compared to standard ViTs.
  arXiv  Detail & Related papers  (2025-11-13T16:56:24Z)
• ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models [7.7352936204066]
  We propose a novel, zero-shot method to model visual token pruning as a balance between task relevance and information diversity.<n>Our method achieves performance that matches or surpasses the state-of-the-art with only minimal accuracy loss.<n>These gains are accompanied by significant reductions in GPU memory footprint and inference latency.
  arXiv  Detail & Related papers  (2025-10-20T06:18:47Z)
• ToDRE: Visual Token Pruning via Diversity and Task Awareness for Efficient Large Vision-Language Models [59.47738955960352]
  ToDRE is a two-stage and training-free token compression framework.<n>It achieves superior performance by pruning tokens based on token Diversity and token-task RElevance.
  arXiv  Detail & Related papers  (2025-05-24T15:47:49Z)
• Token Sequence Compression for Efficient Multimodal Computing [0.19116784879310028]
  The exponential growth of Large Multimodal Models (LMMs) has driven advancements in cross-modal reasoning but at significant computational costs.<n>We highlight the redundancy and inefficiency in current vision encoders, and seek to construct an adaptive compression method for multimodal data.<n>This work is a first effort towards more effective encoding and processing of high-dimensional data, and paves the way for more scalable and sustainable multimodal systems.
  arXiv  Detail & Related papers  (2025-04-24T19:11:10Z)
• "Principal Components" Enable A New Language of Images [79.45806370905775]
  We introduce a novel visual tokenization framework that embeds a provable PCA-like structure into the latent token space.<n>Our approach achieves state-of-the-art reconstruction performance and enables better interpretability to align with the human vision system.
  arXiv  Detail & Related papers  (2025-03-11T17:59:41Z)
• Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters [54.01228554126122]
  Vision Language Models (VLMs) have demonstrated strong capabilities across various visual understanding and reasoning tasks.<n>To reduce inference costs, one can either downsize the Large Language Models (LLMs) or reduce the number of input tokens needed to represent the image.<n>We take the first steps toward designing token compression algorithms tailored for high-compression settings.
  arXiv  Detail & Related papers  (2024-11-05T18:54:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.