Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2607.28341v1
• Date: Thu, 30 Jul 2026 15:07:00 GMT
• Title: Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models
• Abstract summary: We propose Trend-aware Pruning, a novel framework that elevates pruning from a local snapshot decision to a temporal trajectory modeling problem.<n>This enables a dynamic rectification mechanism that selectively reactivates "late-blooming" tokens, those initially undervalued but exhibiting rising semantic importance, thereby preventing the loss of critical visual cues.
• Score: 86.02826269126308
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While visual token pruning is essential for efficient Multimodal Large Language Models (MLLMs), existing training-free methods suffer from a critical limitation: they rely on static, instantaneous heuristics to perform irreversible filtering. This approach ignores the hierarchical nature of MLLMs, where token importance often evolves dynamically rather than remaining fixed across layers. Consequently, tokens essential for deep-layer reasoning are often prematurely discarded by shallow-layer estimates. To address this, we propose Trend-aware Pruning, a novel framework that elevates pruning from a local snapshot decision to a temporal trajectory modeling problem. Instead of relying on isolated scores, our method captures the momentum of attention flow. This enables a dynamic rectification mechanism that selectively reactivates "late-blooming" tokens, those initially undervalued but exhibiting rising semantic importance, thereby preventing the loss of critical visual cues. Extensive experiments demonstrate that our approach achieves a superior efficiency-performance trade-off across diverse multimodal tasks. Notably, it reduces visual tokens by over 77.8%, retaining only approximately 23 tokens in the final layer while maintaining competitive performance, offering a robust and reversible solution for high-efficiency multimodal inference.

Related papers

• One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs [18.48496973561215]
  Large Vision-Language Models (LVLMs) have achieved remarkable success across diverse multimodal tasks, yet their practical deployment remains constrained by the computational burden arising from lengthy visual tokens.<n>We propose Adaptive Layer-wise Visual Token Selection (ALVTS), a novel framework that breaks away from the conventional static token pruning paradigm.<n>With an 89% token compression ratio, ALVTS retains 96.7% of the original model's accuracy, achieving a superior efficiency-accuracy trade-off for LVLM inference.
  arXiv  Detail & Related papers  (2026-06-12T08:58:58Z)
• Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs [89.7106332677868]
  We present Visual-Skip, a training-free inference paradigm that decouples spatial interaction from semantic evolution.<n>V-Skip effectively bypasses redundant vision attention to achieve block-wise sparsity, maintaining a 94.16% to 100.31% performance retention across diverse MLLMs.
  arXiv  Detail & Related papers  (2026-06-07T08:32:13Z)
• A Progressive Training Strategy for Vision-Language Models to Counteract Spatio-Temporal Hallucinations in Embodied Reasoning [49.61652671596548]
  "Multi-image hallucination reasoning", where a massive performance drop between forward and temporal queries reveals a dependence on superficial shortcuts instead of genuine understanding.<n>To mitigate this, we develop a new dataset: Chain-of-temporal building on detailed reasoning into Chain steps and definitive judgments.<n>Our experiments demonstrate that this approach not only improves accuracy but also the forward-backward performance gap from over 70% to only 6.53%.
  arXiv  Detail & Related papers  (2026-04-12T07:48:44Z)
• SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass [20.7003663809766]
  Visual token pruning is a promising approach for reducing the computational cost of vision-language models.<n>We introduce a new pruning paradigm, termed bypass, which preserves unselected visual tokens and forwards them to subsequent pruning stages.<n>Building on this paradigm, we propose SwiftVLM, a simple and training-free method that performs pruning at model-specific layers with strong visual token selection capability.
  arXiv  Detail & Related papers  (2026-02-03T05:42:51Z)
• Forest Before Trees: Latent Superposition for Efficient Visual Reasoning [61.29300723302152]
  Laser is a novel paradigm that reformulates visual deduction via Dynamic Windowed Alignment Learning (DWAL)<n>Laser achieves state-of-the-art performance among latent reasoning methods, surpassing the strong baseline Monet by 5.03% on average.
  arXiv  Detail & Related papers  (2026-01-11T08:30:49Z)
• Training-Free Token Pruning via Zeroth-Order Gradient Estimation in Vision-Language Models [16.540220733551823]
  Large Vision-Language Models (VLMs) enable strong multimodal reasoning but incur heavy inference costs from redundant visual tokens.<n> Attention-based methods rely on raw attention scores, which are often unstable across layers and heads.<n>We propose ours, a training-free framework built on a simple intuition.
  arXiv  Detail & Related papers  (2025-09-29T14:20:05Z)
• GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models [5.025353943896242]
  GreedyPrune is a training-free visual token pruning algorithm designed to optimize semantic saliency and visual diversity.<n>We show that GreedyPrune achieves state-of-the-art accuracy across various multimodal tasks and models while significantly reducing end-to-end inference latency.
  arXiv  Detail & Related papers  (2025-06-16T07:21:11Z)
• Bridging Continuous and Discrete Tokens for Autoregressive Visual Generation [85.82112629564942]
  We propose TokenBridge, which maintains the strong representation capacity of continuous tokens while preserving the modeling simplicity of discrete tokens.<n>We introduce a dimension-wise quantization strategy that independently discretizes each feature dimension, paired with a lightweight autoregressive prediction mechanism.<n>Our approach achieves reconstruction and generation quality on par with continuous methods while using standard categorical prediction.
  arXiv  Detail & Related papers  (2025-03-20T17:59:59Z)
• PRISM: Self-Pruning Intrinsic Selection Method for Training-Free Multimodal Data Selection [68.8373788348678]
  Visual instruction tuning adapts pre-trained Multimodal Large Language Models to follow human instructions.<n>PRISM is the first training-free framework for efficient visual instruction selection.<n>It reduces the end-to-end time for data selection and model tuning to just 30% of conventional pipelines.
  arXiv  Detail & Related papers  (2025-02-17T18:43:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.