Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2606.24165v1
• Date: Tue, 23 Jun 2026 05:39:52 GMT
• Title: Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models
• Abstract summary: We propose a training-free token pruning framework based on Cross-Layer Spectral Evolution (CLSE)<n>Instead of measuring token importance from single-layer feature magnitudes, CLSE quantifies how token representations evolve across Transformer layers in the frequency domain.<n>CLSE achieves a superior trade-off between efficiency and accuracy under aggressive token reduction.
• Score: 46.61944018676346
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reducing visual token redundancy is critical for accelerating Multimodal Large Language Models (MLLMs) without degrading cross-modal reasoning performance. Existing token pruning methods typically rely on single-layer signals, such as attention scores or token similarities, which overlook the cross-layer transformation of visual representations and may exhibit positional bias in multimodal token sequences. To address this limitation, we propose a training-free token pruning framework based on Cross-Layer Spectral Evolution (CLSE). Instead of measuring token importance from single-layer feature magnitudes, CLSE quantifies how token representations evolve across Transformer layers in the frequency domain. This evolution reflects the transition from high-frequency structural details to low-frequency semantic abstractions. We observe that tokens with stronger spectral redistribution across layers are more likely to be semantically active and should therefore be preserved. By modeling cross-layer token dynamics, CLSE provides a stable importance criterion that mitigates positional bias. Extensive experiments on both image and video benchmarks demonstrate that CLSE achieves a superior trade-off between efficiency and accuracy under aggressive token reduction. Across multiple MLLMs, CLSE reduces FLOPs, KV cache memory, and latency while maintaining competitive or improved performance.

Related papers

• Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models [86.02826269126308]
  We propose Trend-aware Pruning, a novel framework that elevates pruning from a local snapshot decision to a temporal trajectory modeling problem.<n>This enables a dynamic rectification mechanism that selectively reactivates "late-blooming" tokens, those initially undervalued but exhibiting rising semantic importance, thereby preventing the loss of critical visual cues.
  arXiv  Detail & Related papers  (2026-07-30T15:07:00Z)
• ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs [77.57757084153083]
  We propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs.<n> ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR)
  arXiv  Detail & Related papers  (2026-06-30T17:20:29Z)
• Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMs [33.66167879532472]
  Multimodal large language models (MLLMs) are increasingly required to adapt to non-stationary streams of visual domains, question types, and user instructions.<n>Existing continual vision-language methods mainly preserve outputs, replay data or pseudo-data, regularize embedding geometry, or allocate task-specific parameters.<n>We propose Attention-Spectrum Regularization (ASR), a replay-free continual learning framework that preserves skill-conditioned structures of cross-modal attention.
  arXiv  Detail & Related papers  (2026-06-22T09:13:53Z)
• One Layer's Trash is Another Layer's Treasure: Adaptive Layer-wise Visual Token Selection in LVLMs [18.48496973561215]
  Large Vision-Language Models (LVLMs) have achieved remarkable success across diverse multimodal tasks, yet their practical deployment remains constrained by the computational burden arising from lengthy visual tokens.<n>We propose Adaptive Layer-wise Visual Token Selection (ALVTS), a novel framework that breaks away from the conventional static token pruning paradigm.<n>With an 89% token compression ratio, ALVTS retains 96.7% of the original model's accuracy, achieving a superior efficiency-accuracy trade-off for LVLM inference.
  arXiv  Detail & Related papers  (2026-06-12T08:58:58Z)
• Look Less, Reason More: Block-wise Attention Skipping for Efficient Multimodal LLMs [89.7106332677868]
  We present Visual-Skip, a training-free inference paradigm that decouples spatial interaction from semantic evolution.<n>V-Skip effectively bypasses redundant vision attention to achieve block-wise sparsity, maintaining a 94.16% to 100.31% performance retention across diverse MLLMs.
  arXiv  Detail & Related papers  (2026-06-07T08:32:13Z)
• RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT [0.014257559724536567]
  Vision Transformers (ViTs) achieve strong performance but suffer from high computational costs due to quadratic self-attention complexity.<n>We propose a depth-aware token reduction framework that adapts reduction strategies to the layer-wise characteristics of token representations.<n>Our framework provides a training-free template for optimizing vision models by aligning reduction strategies with hierarchical feature evolution.
  arXiv  Detail & Related papers  (2026-06-06T13:13:02Z)
• BATR-FST: Bi-Level Adaptive Token Refinement for Few-Shot Transformers [2.5680214354539803]
  We propose Bi-Level Adaptive Token Refinement for Few-Shot Transformers (BATR-FST)<n>BATR-FST progressively improves token representations and maintains a robust inductive bias for few-shot classification.<n>It achieves superior results in both 1-shot and 5-shot scenarios and improves the few-shot classification via transformers.
  arXiv  Detail & Related papers  (2025-09-16T07:33:21Z)
• Causal Disentanglement and Cross-Modal Alignment for Enhanced Few-Shot Learning [11.752632557524969]
  Causal CLIP Adapter (CCA) is a novel framework that explicitly disentangles visual features extracted from CLIP.<n>Our method consistently outperforms state-of-the-art approaches in terms of few-shot performance and robustness to distributional shifts.
  arXiv  Detail & Related papers  (2025-08-05T05:30:42Z)
• TokLIP: Marry Visual Tokens to CLIP for Multimodal Comprehension and Generation [80.90309237362526]
  TokLIP is a visual tokenizer that enhances comprehension by semanticizing vector-quantized (VQ) tokens.<n>TokLIP integrates a low-level discrete VQ tokenizer with a ViT-based token encoder to capture high-level continuous semantics.
  arXiv  Detail & Related papers  (2025-05-08T17:12:19Z)
• Semantic Equitable Clustering: A Simple and Effective Strategy for Clustering Vision Tokens [57.37893387775829]
  We introduce a fast and balanced clustering method, named Semantic Equitable Clustering (SEC)<n>SEC clusters tokens based on their global semantic relevance in an efficient, straightforward manner.<n>We propose a versatile vision backbone, SECViT, to serve as a vision language connector.
  arXiv  Detail & Related papers  (2024-05-22T04:49:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.