SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models

• URL: http://arxiv.org/abs/2608.03580v1
• Date: Tue, 04 Aug 2026 12:34:46 GMT
• Title: SlimVLM: Sensitivity-aware Dynamic Structured Pruning with Adaptive Visual Token Selection for Efficient Vision-Language Models
• Abstract summary: Vision-Language Models (VLMs) have demonstrated remarkable performance in processing and understanding both text and images.<n>We propose SlimVLM, a structured pruning framework designed to compress VLMs while preserving their task performance.
• Score: 13.091215419258857
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Vision-Language Models (VLMs) have demonstrated remarkable performance in processing and understanding both text and images, their large parameter sizes lead to significant computational overhead, limiting their deployment on resource-constrained devices. While pruning has been effective for compressing Large Language Models (LLMs), directly applying it to VLMs leads to significant performance drops, largely due to redundant visual tokens interfering with importance estimation. To this end, we propose SlimVLM, a structured pruning framework designed to compress VLMs while preserving their task performance. We introduce an adaptive visual token selection strategy for VLMs that leverages average text-to-visual attention scores to assess the importance of visual tokens, removing redundant ones during pruning based on a set threshold, thereby optimizing the importance calculation. Recognizing the varying tolerance to sparsity across different modules, we also propose a Sensitivity-aware dynamic pruning mechanism that determines the appropriate pruning ratio for each module by calculating the linear reconstruction error between the outputs of the pruned and unpruned modules, ensuring overall performance stability. Experimental results show that SlimVLM outperforms existing methods across multiple multimodal benchmarks, achieving state-of-the-art performance.

Related papers

• VisPCO: Visual Token Pruning Configuration Optimization via Budget-Aware Pareto-Frontier Learning for Vision-Language Models [74.05545957865228]
  Visual token pruning methods effectively mitigate the quadratic computational growth caused by processing high-resolution images and video frames in vision-language models (VLMs)<n>We introduce, a novel framework that formulates visual token pruning as a Pareto configuration optimization problem to automatically identify optimal configurations.<n>Our approach employs continuous relaxation and straight-through estimators to enable gradient-based search, solved via the Augmented Lagrangian method.
  arXiv  Detail & Related papers  (2026-04-16T16:21:05Z)
• Learning What Matters: Dynamic Dimension Selection and Aggregation for Interpretable Vision-Language Reward Modeling [14.249938992666202]
  VL-MDR (Vision-Language Multi-Dimensional Reward) is a framework that decomposes evaluation into granular, interpretable dimensions.<n>To support this, we curate a dataset of 321k vision-language preference pairs annotated across 21 fine-grained dimensions.
  arXiv  Detail & Related papers  (2026-04-07T05:28:39Z)
• Empirical Recipes for Efficient and Compact Vision-Language Models [54.92440500651415]
  Vision-language models (VLMs) in resource-constrained settings demand low latency and high throughput.<n>We conduct an empirical end-to-end efficiency analysis and systematically profile inference to identify the dominant bottlenecks.<n>We develop optimization recipes tailored to compact VLMs that substantially reduce latency while preserving accuracy.
  arXiv  Detail & Related papers  (2026-03-17T17:17:40Z)
• MURE: Hierarchical Multi-Resolution Encoding via Vision-Language Models for Visual Document Retrieval [111.99106496142474]
  Visual Document Retrieval (VDR) requires representations that capture both fine-grained visual details and global document structure.<n>Existing VDR models struggle to balance effectiveness and efficiency when processing high-resolution documents.<n>We propose a new X-VisEmb paradigm that progresses from multi-resolution sampling and encoding, through cross-granularity feature fusion, to adaptive representation distillation.
  arXiv  Detail & Related papers  (2026-03-07T15:17:22Z)
• Input-Adaptive Visual Preprocessing for Efficient Fast Vision-Language Model Inference [2.8292841621378844]
  We propose an adaptive visual preprocessing method that adjusts input resolution and spatial coverage based on image content characteristics.<n>The proposed approach combines content-aware image analysis, adaptive resolution selection, and content-aware cropping to reduce visual redundancy prior to vision encoding.<n> Experimental results show that adaptive preprocessing reduces per-image inference time by over 50%, mean full generation time, and achieves a consistent reduction of more than 55% in visual token count.
  arXiv  Detail & Related papers  (2025-12-23T23:30:56Z)
• DyMU: Dynamic Merging and Virtual Unmerging for Efficient VLMs [124.52164183968145]
  We present DyMU, an efficient, training-free framework that reduces the computational burden of vision-language models (VLMs)<n>Our approach comprises two key components. First, Dynamic Token Merging (DToMe) reduces the number of visual token embeddings by merging similar tokens based on image complexity.<n>Second, Virtual Token Unmerging (VTU) simulates the expected token sequence for large language models (LLMs) by efficiently reconstructing the attention dynamics of a full sequence.
  arXiv  Detail & Related papers  (2025-04-23T18:38:18Z)
• ContextFormer: Redefining Efficiency in Semantic Segmentation [48.81126061219231]
  Convolutional methods, although capturing local dependencies well, struggle with long-range relationships.<n>Vision Transformers (ViTs) excel in global context capture but are hindered by high computational demands.<n>We propose ContextFormer, a hybrid framework leveraging the strengths of CNNs and ViTs in the bottleneck to balance efficiency, accuracy, and robustness for real-time semantic segmentation.
  arXiv  Detail & Related papers  (2025-01-31T16:11:04Z)
• Selective State Space Memory for Large Vision-Language Models [0.0]
  State Space Memory Integration (SSMI) is a novel approach for efficient fine-tuning of LVLMs.<n>SSMI captures long-range dependencies and injects task-specific visual and sequential patterns effectively.<n> experiments on benchmark datasets, including COCO Captioning, VQA, and Flickr30k, demonstrate that SSMI achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-13T05:40:50Z)
• Efficient High-Resolution Visual Representation Learning with State Space Model for Human Pose Estimation [60.80423207808076]
  Capturing long-range dependencies while preserving high-resolution visual representations is crucial for dense prediction tasks such as human pose estimation.<n>We propose the Dynamic Visual State Space (DVSS) block, which augments visual state space models with multi-scale convolutional operations.<n>We build HRVMamba, a novel model for efficient high-resolution representation learning.
  arXiv  Detail & Related papers  (2024-10-04T06:19:29Z)
• SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Models [35.5601603013045]
  We propose SmartTrim, an adaptive acceleration framework for Vision-Language Models (VLMs) We integrate lightweight modules into the original backbone to identify and prune redundant token representations and attention heads within each layer. We devise a self-distillation strategy to enhance the consistency between the predictions of the pruned model and its fully-capacity counterpart.
  arXiv  Detail & Related papers  (2023-05-24T11:18:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.