SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models

• URL: http://arxiv.org/abs/2607.25818v1
• Date: Tue, 28 Jul 2026 15:01:02 GMT
• Title: SepPrune:A Separator-based Pruning Framework for Efficient Multimodal Large Language Models
• Abstract summary: We propose SepPrune, an efficient, training-free, plug-and-play pruning method for vision tokens.<n>SepPrune reuses the LLM's built-in projection parameters and requires no architectural changes.<n> Experiments on Qwen2.5-VL-7B show that SepPrune achieves state-of-the-art performance, retaining 96.3% of the original accuracy while removing 80.2% of vision tokens.
• Score: 22.517061458809057
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent multimodal large language models (MLLMs), such as Qwen2.5-VL and InternVL3, generate large numbers of vision tokens for high-resolution inputs, leading to substantial computational cost. Existing vision token pruning methods either depend on cross-modal attention and cannot prune before the prefill stage, or rely on diversity estimation with high computational overhead. We observe that attention scores from both vision and text tokens peak at modality separator tokens, suggesting that these separators bridge the two modalities. Based on this observation, we propose SepPrune, an efficient, training-free, plug-and-play pruning method that uses the separator token as a unified query to rank and select informative vision tokens. SepPrune reuses the LLM's built-in projection parameters and requires no architectural changes. Experiments on Qwen2.5-VL-7B show that SepPrune achieves state-of-the-art performance, retaining 96.3% of the original accuracy while removing 80.2% of vision tokens.

Related papers

• STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models [54.273344678561024]
  Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose computational overhead.<n>We conduct two complementary analyses of visual token pruning.<n>We propose STAR-Pro, a training-free two-stage framework for visual token pruning.
  arXiv  Detail & Related papers  (2026-09-05T06:18:28Z)
• TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference [54.41143483594129]
  Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens.<n>Visual token pruning offers a natural solution, yet existing methods are imperfect.<n>We propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs.
  arXiv  Detail & Related papers  (2026-06-25T15:29:37Z)
• Stepwise Token Selection for Efficient Multimodal Large Language Models [0.7233065479782755]
  We introduce a pointer-style selection mechanism that iteratively chooses informative tokens, conditioning each decision on previously selected ones, and dynamically determines when to stop.<n>Under aggressive pruning that removes 88.9% of visual tokens, our method preserves 94.6% of the original accuracy while achieving a 1.88x speed-up in prefill latency.
  arXiv  Detail & Related papers  (2026-06-14T23:50:08Z)
• IDPruner: Harmonizing Importance and Diversity in Visual Token Pruning for MLLMs [11.254129271889035]
  Visual token pruning has emerged as a critical technique for accelerating MLLM inference.<n>IDPruner achieves state-of-the-art performance and superior generalization across diverse architectures and tasks.
  arXiv  Detail & Related papers  (2026-02-10T11:20:24Z)
• D2Pruner: Debiased Importance and Structural Diversity for MLLM Token Pruning [49.16227597771663]
  D2Pruner is a framework that combines debiased importance with a structural pruning mechanism.<n>It reduces FLOPs by 74.2% while retaining 99.2% of its original performance.<n>It marks a significant advancement with up to 63. 53% improvement over existing methods.
  arXiv  Detail & Related papers  (2025-12-22T14:42:31Z)
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs [59.415473779171315]
  We propose a novel visual token pruning strategy called textbfSaliency-textbfCoverage textbfOriented token textbfPruning for textbfEfficient MLLMs.
  arXiv  Detail & Related papers  (2025-10-28T09:29:37Z)
• CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models [75.88232735646018]
  Large Vision-Language Models (LVLMs) process multimodal inputs consisting of text tokens and vision tokens extracted from images or videos.<n>Existing methods attempt to prune redundant vision tokens, revealing substantial redundancy in visual representations.<n>We propose CoViPAL, a layer-wise contextualized visual token pruning method that employs a Plug-and-Play Pruning Module (PPM) to predict and remove redundant vision tokens before they are processed by the LVLM.
  arXiv  Detail & Related papers  (2025-08-24T07:47:00Z)
• Beyond Attention or Similarity: Maximizing Conditional Diversity for Token Pruning in MLLMs [30.97955016203357]
  In multimodal large language models, the length of input visual tokens is often significantly greater than that of their textual counterparts.<n>We propose a novel visual token pruning method named CDPruner, which maximizes the conditional diversity of retained tokens.<n>Our experiments show that CDPruner establishes new state-of-the-art on various vision-based benchmarks.
  arXiv  Detail & Related papers  (2025-06-12T17:59:09Z)
• Balanced Token Pruning: Accelerating Vision Language Models Beyond Local Optimization [30.73986620551153]
  Large Vision-Language Models (LVLMs) have shown impressive performance across multi-modal tasks by encoding images into thousands of tokens.<n>Previous approaches have attempted to reduce the number of image tokens through token pruning.<n>We propose Balanced Token Pruning (BTP), a plug-and-play method for pruning vision tokens.
  arXiv  Detail & Related papers  (2025-05-28T07:00:50Z)
• Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models [85.51753014478315]
  We introduce AdaptPrune, a novel plug-and-play training-free pruning method.<n>It builds on conventional attention-based pruning by integrating spatial distance and token similarity with an adaptive NMS approach.<n>Our approach ensures a comprehensive evaluation of token importance and substantially refines the pruning decisions.
  arXiv  Detail & Related papers  (2025-03-11T03:58:17Z)
• Sparsity Meets Similarity: Leveraging Long-Tail Distribution for Dynamic Optimized Token Representation in Multimodal Large Language Models [6.467840081978855]
  multimodal large language models (MM-LLMs) have achieved significant success in various tasks.<n>Main computational burden arises from processingd text and visual tokens.<n>We propose a dynamic pruning algorithm that identifies the inflection point in the visual CLS token similarity curve.
  arXiv  Detail & Related papers  (2024-09-02T10:49:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.