GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models

• URL: http://arxiv.org/abs/2506.13166v1
• Date: Mon, 16 Jun 2025 07:21:11 GMT
• Title: GreedyPrune: Retenting Critical Visual Token Set for Large Vision Language Models
• Abstract summary: GreedyPrune is a training-free visual token pruning algorithm designed to optimize semantic saliency and visual diversity.<n>We show that GreedyPrune achieves state-of-the-art accuracy across various multimodal tasks and models while significantly reducing end-to-end inference latency.
• Score: 5.025353943896242
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Although Large Vision Language Models (LVLMs) have demonstrated remarkable performance in image understanding tasks, their computational efficiency remains a significant challenge, particularly on resource-constrained devices due to the high cost of processing large numbers of visual tokens. Recently, training-free visual token pruning methods have gained popularity as a low-cost solution to this issue. However, existing approaches suffer from two key limitations: semantic saliency-based strategies primarily focus on high cross-attention visual tokens, often neglecting visual diversity, whereas visual diversity-based methods risk inadvertently discarding semantically important tokens, especially under high compression ratios. In this paper, we introduce GreedyPrune, a training-free plug-and-play visual token pruning algorithm designed to jointly optimize semantic saliency and visual diversity. We formalize the token pruning process as a combinatorial optimization problem and demonstrate that greedy algorithms effectively balance computational efficiency with model accuracy. Extensive experiments validate the effectiveness of our approach, showing that GreedyPrune achieves state-of-the-art accuracy across various multimodal tasks and models while significantly reducing end-to-end inference latency.

Related papers

• Capturing Token Tendencies for Training-Free Token Pruning in Multimodal Large Language Models [86.02826269126308]
  We propose Trend-aware Pruning, a novel framework that elevates pruning from a local snapshot decision to a temporal trajectory modeling problem.<n>This enables a dynamic rectification mechanism that selectively reactivates "late-blooming" tokens, those initially undervalued but exhibiting rising semantic importance, thereby preventing the loss of critical visual cues.
  arXiv  Detail & Related papers  (2026-07-30T15:07:00Z)
• VISion On Request: Enhanced VLLM efficiency with sparse, dynamically selected, vision-language interactions [51.41587958253802]
  We introduce VISion On Request (VISOR), a method that reduces inference cost without discarding visual information.<n>VISOR improves efficiency by sparsifying the interaction between image and text tokens.<n>Experiments show that VISOR drastically reduces computational cost while matching or exceeding state-of-the-art results.
  arXiv  Detail & Related papers  (2026-03-24T17:58:17Z)
• ViTCoP: Accelerating Large Vision-Language Models via Visual and Textual Semantic Collaborative Pruning [8.933549837045932]
  Large Vision-Language Models incur high computational costs due to significant redundancy in their visual tokens.<n>We propose a Visual and Textual Semantic Collaborative Pruning framework (ViTCoP) that combines redundancy filtering in the vision encoder with step-wise co-pruning within the Large Language Models.
  arXiv  Detail & Related papers  (2026-01-25T12:47:30Z)
• ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models [7.7352936204066]
  We propose a novel, zero-shot method to model visual token pruning as a balance between task relevance and information diversity.<n>Our method achieves performance that matches or surpasses the state-of-the-art with only minimal accuracy loss.<n>These gains are accompanied by significant reductions in GPU memory footprint and inference latency.
  arXiv  Detail & Related papers  (2025-10-20T06:18:47Z)
• ToDRE: Visual Token Pruning via Diversity and Task Awareness for Efficient Large Vision-Language Models [59.47738955960352]
  ToDRE is a two-stage and training-free token compression framework.<n>It achieves superior performance by pruning tokens based on token Diversity and token-task RElevance.
  arXiv  Detail & Related papers  (2025-05-24T15:47:49Z)
• Top-Down Compression: Revisit Efficient Vision Token Projection for Visual Instruction Tuning [70.57180215148125]
  Visual instruction tuning aims to enable large language models to comprehend the visual world.<n>Existing methods often grapple with the intractable trade-off between accuracy and efficiency.<n>We present LLaVA-Meteor, a novel approach that strategically compresses visual tokens without compromising core information.
  arXiv  Detail & Related papers  (2025-05-17T10:22:29Z)
• TopV: Compatible Token Pruning with Inference Time Optimization for Fast and Low-Memory Multimodal Vision Language Model [56.43860351559185]
  We introduce textbfTopV, a compatible textbfTOken textbfPruning with inference Time Optimization for fast and low-memory textbfVLM.<n>Our framework incorporates a visual-aware cost function to measure the importance of each source visual token, enabling effective pruning of low-importance tokens.
  arXiv  Detail & Related papers  (2025-03-24T01:47:26Z)
• Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models [85.51753014478315]
  We introduce AdaptPrune, a novel plug-and-play training-free pruning method.<n>It builds on conventional attention-based pruning by integrating spatial distance and token similarity with an adaptive NMS approach.<n>Our approach ensures a comprehensive evaluation of token importance and substantially refines the pruning decisions.
  arXiv  Detail & Related papers  (2025-03-11T03:58:17Z)
• Learning Free Token Reduction for Multi-Modal Large Language Models [3.4026156483879517]
  Vision-Language Models (VLMs) have achieved remarkable success across a range of multimodal tasks.<n>However, their practical deployment is often constrained by high computational costs and prolonged inference times.<n>We propose a token compression paradigm that operates on both spatial and temporal dimensions.
  arXiv  Detail & Related papers  (2025-01-29T02:52:32Z)
• Efficient Multi-modal Large Language Models via Visual Token Grouping [55.482198808206284]
  High-resolution images and videos pose a barrier to their broader adoption.<n> compressing vision tokens in MLLMs has emerged as a promising approach to reduce inference costs.<n>We introduce VisToG, a novel grouping mechanism that leverages the capabilities of pre-trained vision encoders to group similar image segments.
  arXiv  Detail & Related papers  (2024-11-26T09:36:02Z)
• FocusLLaVA: A Coarse-to-Fine Approach for Efficient and Effective Visual Token Compression [45.37530855889661]
  High-resolution images lead to a quadratic increase in the number of visual tokens input into Multi-modal Large Language Models. Current work develop visual token compression methods to achieve efficiency improvements, often at the expense of performance. We build a coarse-to-fine visual token compression method, with a vision-guided sampler for compressing redundant regions with low information density, and a text-guided sampler for selecting visual tokens that are strongly correlated with the user instructions.
  arXiv  Detail & Related papers  (2024-11-21T15:37:52Z)
• Inference Optimal VLMs Need Fewer Visual Tokens and More Parameters [54.01228554126122]
  Vision Language Models (VLMs) have demonstrated strong capabilities across various visual understanding and reasoning tasks.<n>To reduce inference costs, one can either downsize the Large Language Models (LLMs) or reduce the number of input tokens needed to represent the image.<n>We take the first steps toward designing token compression algorithms tailored for high-compression settings.
  arXiv  Detail & Related papers  (2024-11-05T18:54:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.