HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models

• URL: http://arxiv.org/abs/2604.07812v1
• Date: Thu, 09 Apr 2026 05:09:22 GMT
• Title: HAWK: Head Importance-Aware Visual Token Pruning in Multimodal Models
• Abstract summary: In multimodal large language models (MLLMs), the surge of visual tokens significantly increases the inference time and computational overhead.<n>Visual token pruning is a promising strategy for reducing the cost of MLLM inference by removing redundant visual tokens.<n>Hawk is a head importance-aware visual token pruning method that perceives the varying importance of attention heads in visual tasks to maximize the retention of crucial tokens.
• Score: 41.41768757204328
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In multimodal large language models (MLLMs), the surge of visual tokens significantly increases the inference time and computational overhead, making them impractical for real-time or resource-constrained applications. Visual token pruning is a promising strategy for reducing the cost of MLLM inference by removing redundant visual tokens. Existing research usually assumes that all attention heads contribute equally to the visual interpretation. However, our study reveals that different heads may capture distinct visual semantics and inherently play distinct roles in visual processing. In light of this observation, we propose HAWK, a head importance-aware visual token pruning method that perceives the varying importance of attention heads in visual tasks to maximize the retention of crucial tokens. By leveraging head importance weights and text-guided attention to assess visual token significance, HAWK effectively retains task-relevant visual tokens while removing redundant ones. The proposed HAWK is entirely training-free and can be seamlessly applied to various MLLMs. Extensive experiments on multiple mainstream vision-language benchmarks demonstrate that HAWK achieves state-of-the-art accuracy. When applied to Qwen2.5-VL, HAWK retains 96.0% of the original accuracy after pruning 80.2% of the visual tokens. Additionally, it reduces end-to-end latency to 74.4% of the original and further decreases GPU memory usage across the tested models. The code is available at https://github.com/peppery77/HAWK.git.

Related papers

• SinkPruner: Sink-Free Visual Token Pruning for Multimodal Large Language Models [25.951841702056544]
  multimodal large language models (MLLMs) incur substantial computational overhead when processing long visual token sequences.<n>We propose SinkPruner, a training-free visual token pruning framework for efficient MLLM inference.<n>SinkPruner follows a coarse-to-fine design with two key modules: a visual sanitizer that filters high-norm redundancies and alleviates attention sink and attention dispersion, and a text-guided pruner that further retains tokens semantically aligned with the text query.
  arXiv  Detail & Related papers  (2026-09-01T09:52:10Z)
• Not All Attention Heads Contribute to Critical Visual Token Selection: Head-Aware Pruning Matters More [10.64705408079015]
  Vision-Language Models (VLMs) have exhibited impressive performance across diverse visual scenarios.<n>To improve VLM inference efficiency, a typical class of visual token pruning methods estimates token importance.<n>We propose ProViP, a training-free progressive visual token pruning framework.
  arXiv  Detail & Related papers  (2026-08-26T03:27:14Z)
• All You Need Are Random Visual Tokens? Demystifying Token Pruning in VLLMs [43.80391827200227]
  In deep layers, existing training-free pruning methods perform no better than random pruning.<n>Visual tokens progressively lose their salience with increasing network depth.<n>We show that simple random pruning in deep layers efficiently balances performance and efficiency.
  arXiv  Detail & Related papers  (2025-12-08T14:16:01Z)
• Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention [50.97683288777336]
  Multimodal Large Language Models (MLLMs) suffer from considerable computational overhead due to their reliance on massive visual tokens.<n>Recent studies have explored token pruning to alleviate this problem, which typically uses text-vision cross-attention.<n>We propose HoloV, a plug-and-play visual token pruning framework for efficient inference.
  arXiv  Detail & Related papers  (2025-10-03T11:33:40Z)
• VFlowOpt: A Token Pruning Framework for LMMs with Visual Information Flow-Guided Optimization [70.98122339799218]
  Large Multimodal Models (LMMs) excel in visual-language tasks by leveraging numerous visual tokens for fine-grained visual information.<n>Previous research aimed at reducing visual tokens during inference typically leverages importance maps derived from attention scores among vision-only tokens or vision-language tokens to prune tokens across one or multiple pruning stages.<n>We propose VFlowOpt, a token pruning framework that introduces an importance map derivation process and a progressive pruning module with a recycling mechanism.<n> Experiments demonstrate that VFlowOpt can prune 90% of visual tokens while maintaining comparable performance, leading to an 89% reduction in KV-Cache memory and 3.8
  arXiv  Detail & Related papers  (2025-08-07T09:47:21Z)
• VisionThink: Smart and Efficient Vision Language Model via Reinforcement Learning [95.89543460132413]
  Vision-language models (VLMs) have improved performance by increasing the number of visual tokens.<n>However, most real-world scenarios do not require such an extensive number of visual tokens.<n>We present a new paradigm for visual token compression, namely, VisionThink.
  arXiv  Detail & Related papers  (2025-07-17T17:59:55Z)
• ToDRE: Visual Token Pruning via Diversity and Task Awareness for Efficient Large Vision-Language Models [59.47738955960352]
  ToDRE is a two-stage and training-free token compression framework.<n>It achieves superior performance by pruning tokens based on token Diversity and token-task RElevance.
  arXiv  Detail & Related papers  (2025-05-24T15:47:49Z)
• Introducing Visual Perception Token into Multimodal Large Language Model [53.82301522384719]
  Multimodal Large Language Model (MLLM) relies on the perception process of its vision encoder.<n>MLLM still lacks the autonomous capability to control its own visual perception processes.<n>We propose the concept of Visual Perception Token, aiming to empower MLLM with a mechanism to control its visual perception processes.
  arXiv  Detail & Related papers  (2025-02-24T18:56:12Z)
• Beyond Text-Visual Attention: Exploiting Visual Cues for Effective Token Pruning in VLMs [34.3615740255575]
  Large vision-language models (LVLMs) generally contain significantly more visual tokens than their textual counterparts.<n>We propose VisPruner, a plug-and-play method that utilizes visual cues for more effective token pruning in LVLMs.<n>Our results show that VisPruner can reduce the FLOPs of LLaVA-1.5-7B by 91% and inference latency by 75%, while maintaining comparable performance.
  arXiv  Detail & Related papers  (2024-12-02T18:57:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.