ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMs

• URL: http://arxiv.org/abs/2603.24680v2
• Date: Tue, 31 Mar 2026 17:09:40 GMT
• Title: ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMs
• Abstract summary: ReDiPrune is a training-free token pruning method applied before the vision-conditioned projector.<n>It selects informative tokens directly from vision encoder outputs, preserving fine-grained spatial and semantic cues.<n>It consistently improves the accuracy-efficiency trade-off across four video and five image benchmarks.
• Score: 16.523460406504604
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent multimodal large language models are computationally expensive because Transformers must process a large number of visual tokens. We present ReDiPrune, a training-free token pruning method applied before the vision-language projector, where visual features remain rich and discriminative. Unlike post-projection pruning methods that operate on compressed representations, ReDiPrune selects informative tokens directly from vision encoder outputs, preserving fine-grained spatial and semantic cues. Each token is scored by a lightweight rule that jointly consider text-conditioned relevance and max-min diversity, ensuring the selected tokens are both query-relevant and non-redundant. ReDiPrune is fully plug-and-play, requiring no retraining or architectural modifications, and can be seamlessly inserted between the encoder and projector. Across four video and five image benchmarks, it consistently improves the accuracy-efficiency trade-off. For example, on EgoSchema with LLaVA-NeXT-Video-7B, retaining only 15% of visual tokens yields a +2.0% absolute accuracy gain while reducing computation by more than $6\times$ in TFLOPs. Code is available at https://github.com/UA-CVML/ReDiPrune.

Related papers

• A More Word-like Image Tokenization for MLLMs [26.120899392740203]
  We propose a novel Disentangled Visual Tokenization (DiVT) that clusters patch embeddings into coherent semantic units.<n>Across diverse multimodal benchmarks, DiVT matches or surpasses baselines with significantly fewer visual tokens.
  arXiv  Detail & Related papers  (2026-05-18T07:09:46Z)
• ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models [59.94664910790462]
  ResPrune is a training-free visual token pruning framework for large vision-language models.<n>It selects a compact yet informative subset of visual tokens.<n>It achieves effective reductions in computation, memory consumption, and inference latency.
  arXiv  Detail & Related papers  (2026-03-22T07:44:45Z)
• PIO-FVLM: Rethinking Training-Free Visual Token Reduction for VLM Acceleration from an Inference-Objective Perspective [59.24570811503256]
  We propose PIO-FVLM to reduce redundant visual tokens in vision-models (VLMs) to accelerate inference.<n>The proposed PIO-FVLM is training-free and compatible with FlashAttention, friendly to practical application and deployment.<n>On LLaVA-Next-7B, PIO-FVLM retains just 11.1% of visual tokens but maintains 97.2% of the original performance.
  arXiv  Detail & Related papers  (2026-02-04T15:33:10Z)
• IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning [27.75049214892312]
  Large Vision-Language Models (LVLMs) achieve impressive performance across multiple tasks.<n>A significant challenge, however, is their prohibitive inference cost when processing high-resolution visual inputs.<n>We propose textbfIVC-Prune, a training-free, prompt-aware pruning strategy that retains both IVC tokens and semantically relevant foreground tokens.
  arXiv  Detail & Related papers  (2026-02-03T03:39:31Z)
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs [59.415473779171315]
  We propose a novel visual token pruning strategy called textbfSaliency-textbfCoverage textbfOriented token textbfPruning for textbfEfficient MLLMs.
  arXiv  Detail & Related papers  (2025-10-28T09:29:37Z)
• H$_{2}$OT: Hierarchical Hourglass Tokenizer for Efficient Video Pose Transformers [124.11648300910444]
  We present a hierarchical plug-and-play pruning-and-$-recovering framework, called Hierarchical Hourglass Tokenizer (H$_2$OT)<n>Our method is general-purpose: it can be easily incorporated into common VPT models on both seq2seq and seq2frame pipelines.
  arXiv  Detail & Related papers  (2025-09-08T17:59:59Z)
• CoViPAL: Layer-wise Contextualized Visual Token Pruning for Large Vision-Language Models [75.88232735646018]
  Large Vision-Language Models (LVLMs) process multimodal inputs consisting of text tokens and vision tokens extracted from images or videos.<n>Existing methods attempt to prune redundant vision tokens, revealing substantial redundancy in visual representations.<n>We propose CoViPAL, a layer-wise contextualized visual token pruning method that employs a Plug-and-Play Pruning Module (PPM) to predict and remove redundant vision tokens before they are processed by the LVLM.
  arXiv  Detail & Related papers  (2025-08-24T07:47:00Z)
• DivPrune: Diversity-based Visual Token Pruning for Large Multimodal Models [13.519389777060226]
  Adding visual tokens to Large Multimodal Models (LMMs) increases the total token count, often by thousands.<n>To address this issue, token pruning methods, which remove part of the visual tokens, are proposed.<n>The proposed method, DivPrune, reduces redundancy and achieves the highest diversity of the selected tokens.
  arXiv  Detail & Related papers  (2025-03-04T01:33:14Z)
• What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph [15.364317811275344]
  We propose a graph-based method towards training-free visual token pruning, termed G-Prune.<n>G-Prune regards visual tokens as nodes, and construct their connections based on their semantic similarities.<n>Experiment results show that G-Prune can greatly reduce computation overhead while retaining high performance on both coarse- and fine-grained tasks.
  arXiv  Detail & Related papers  (2025-01-04T12:14:42Z)
• Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters [54.01228554126122]
  Vision Language Models (VLMs) have demonstrated strong capabilities across various visual understanding and reasoning tasks.<n>To reduce inference costs, one can either downsize the Large Language Models (LLMs) or reduce the number of input tokens needed to represent the image.<n>We take the first steps toward designing token compression algorithms tailored for high-compression settings.
  arXiv  Detail & Related papers  (2024-11-05T18:54:21Z)
• SparseVLM: Visual Token Sparsification for Efficient Vision-Language Model Inference [45.11612407862277]
  In vision-language models (VLMs), visual tokens usually bear a significant amount of computational overhead despite sparsity of information in them when compared to text tokens.<n>We propose a text-guided training-free token optimization mechanism dubbed SparseVLM that eliminates the need of extra parameters or fine-tuning costs.
  arXiv  Detail & Related papers  (2024-10-06T09:18:04Z)
• LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models [35.88374542519597]
  Large Multimodal Models (LMMs) have shown significant visual reasoning capabilities by connecting a visual encoder and a large language model. Recent LMMs incorporate more complex visual inputs, such as high-resolution images and videos, which further increases the number of visual tokens significantly. We propose PruMerge, a novel adaptive visual token reduction strategy that significantly reduces the number of visual tokens without compromising the performance of LMMs.
  arXiv  Detail & Related papers  (2024-03-22T17:59:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.