HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment

• URL: http://arxiv.org/abs/2608.23921v1
• Date: Mon, 24 Aug 2026 23:55:36 GMT
• Title: HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment
• Abstract summary: Existing methods score each visual token by averaging text-to-visual attention uniformly across all heads.<n> misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues.<n>We propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions.
• Score: 76.68875424389057
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent Vision-Language Models encode high-resolution images into long visual token sequences, incurring prohibitive prefill costs. To compress them, existing methods score each visual token by averaging text-to-visual attention uniformly across all heads, which assumes every head matches the query. However, our empirical analysis shows that misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues. To address this, we propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions. Built on PAQ, our pruning proceeds in three stages. Given a target FLOPs budget, we first partition the transformer layers into groups and allocate a visual token budget to each. Within each group, we then aggregate per-head attention maps via PAQ-weighted softmax into a group-level matrix. Finally, we score visual tokens by this matrix's magnitude and retain the allocated budget per group. By weighting heads with PAQ, our method scores tokens by attention signals that more faithfully reflect prompt relevance, rather than diluting them through uniform averaging. Across 18 benchmarks, our method delivers state-of-the-art trade-offs. Specifically, on LLaVA-1.5-7B (9 tasks), retaining only \textbf{5.6\%} tokens preserves \textbf{99.1\%} of the original performance, surpassing the strongest baseline AutoPrune by 4.2 points. Code is available in https://github.com/baokou-fw2/HAP.

Related papers

• STAR-Pro: Stage-Wise Token Adaptive Reduction with Progressive Refinement for Efficient Large Vision-Language Models [54.273344678561024]
  Large vision-language models (LVLMs) achieve strong multimodal understanding, but the hundreds to thousands of visual tokens they process impose computational overhead.<n>We conduct two complementary analyses of visual token pruning.<n>We propose STAR-Pro, a training-free two-stage framework for visual token pruning.
  arXiv  Detail & Related papers  (2026-09-05T06:18:28Z)
• From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankers [16.546400103214715]
  Existing pruning methods retain tokens by attention saliency, yet we show that saliency is systematically misaligned with ranking contribution.<n>We propose RaDiCal, a training-free framework that uses normalized attention entropy to decide when saliency can be trusted.<n>It cuts FLOPs by 39--45% and delivers 1.28--1.45$times$ measured speedups across two VLM architectures without dataset-specific retuning.
  arXiv  Detail & Related papers  (2026-09-01T03:45:21Z)
• LAST: The Last Query Token Guides Visual Token Pruning for Edge-Cloud Collaborative MLLM Inference [63.932718076408584]
  LAST is a training-free framework for query-dependent visual token pruning in edge-cloud collaborative MLLM inference.<n>It preserves 95.4% of the full-token accuracy while retaining only 12.5% of the visual tokens, with low edge-side selection overhead and reduced cloud-side computation.
  arXiv  Detail & Related papers  (2026-07-30T09:59:25Z)
• AdaTok: Self-Budgeting Image Tokenization with Quality-Preserving Dynamic Tokens [39.0104982235623]
  We propose AdaTok, a self-budgeting discrete 1D tokenizer.<n>AdaTok combines Prioritized Representation Learning, which orders tokens with nested tail masking, with Adaptive Token Allocation.<n>On ImageNet-1K, AdaTok-Full reaches rFID 1.31 at 256 tokens, while AdaTok-Adaptive attains rFID 1.50 using only 118 tokens on average, outperforming discrete 1D baselines at comparable budgets.
  arXiv  Detail & Related papers  (2026-06-05T11:49:28Z)
• Beyond Attention Magnitude: Leveraging Inter-layer Rank Consistency for Efficient Vision-Language-Action Models [75.27089289058613]
  We introduce textbfTIES (textbfTau-guided textbfInter-layer textbfEfficient textbfSelection), a dynamic framework guided by inter-layer token ranking consistency.<n>On the CogACT + SIMPLER benchmark, TIES improves average success rates by 6% while reducing token usage by 78%, and demonstrate strong generalization across diverse decoders and benchmarks.
  arXiv  Detail & Related papers  (2026-03-26T02:13:03Z)
• All You Need Are Random Visual Tokens? Demystifying Token Pruning in VLLMs [43.80391827200227]
  In deep layers, existing training-free pruning methods perform no better than random pruning.<n>Visual tokens progressively lose their salience with increasing network depth.<n>We show that simple random pruning in deep layers efficiently balances performance and efficiency.
  arXiv  Detail & Related papers  (2025-12-08T14:16:01Z)
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs [59.415473779171315]
  We propose a novel visual token pruning strategy called textbfSaliency-textbfCoverage textbfOriented token textbfPruning for textbfEfficient MLLMs.
  arXiv  Detail & Related papers  (2025-10-28T09:29:37Z)
• Don't Just Chase "Highlighted Tokens" in MLLMs: Revisiting Visual Holistic Context Retention [50.97683288777336]
  Multimodal Large Language Models (MLLMs) suffer from considerable computational overhead due to their reliance on massive visual tokens.<n>Recent studies have explored token pruning to alleviate this problem, which typically uses text-vision cross-attention.<n>We propose HoloV, a plug-and-play visual token pruning framework for efficient inference.
  arXiv  Detail & Related papers  (2025-10-03T11:33:40Z)
• VISA: Group-wise Visual Token Selection and Aggregation via Graph Summarization for Efficient MLLMs Inference [76.00113788838334]
  Group-wise textbfVIsual token textbfSelection and textbfAggregation (VISA)<n>Our method can preserve more visual information while compressing visual tokens.<n>We conduct comprehensive experiments on LLaVA-1.5, LLaVA-NeXT, and Video-LLaVA across various benchmarks to validate the efficacy of VISA.
  arXiv  Detail & Related papers  (2025-08-25T10:07:07Z)
• CompressKV: Semantic Retrieval Heads Know What Tokens are Not Important Before Generation [7.119276797399788]
  Increasing key-value (KV) cache size poses critical challenges to memory and execution efficiency.<n>Most KV cache compression methods rely on token eviction using all attention heads in Grouped Query Attention (GQA)-based LLMs.<n>We introduce a layer-adaptive KV cache allocation strategy, which consistently outperforms state-of-the-art approaches under various memory budgets.
  arXiv  Detail & Related papers  (2025-08-04T13:26:16Z)
• HiPrune: Training-Free Visual Token Pruning via Hierarchical Attention in Vision-Language Models [6.306822764683807]
  HiPrune is a training-free and model-agnostic token Pruning framework for vision encoders.<n>It exploits the Hierarchical attention structure within vision encoders.<n>It preserves up to 99.3% task accuracy with only 33.3% tokens, and maintaining 99.5% accuracy with just 11.1% tokens.
  arXiv  Detail & Related papers  (2025-08-01T11:48:11Z)
• ToSA: Token Selective Attention for Efficient Vision Transformers [50.13756218204456]
  ToSA is a token selective attention approach that can identify tokens that need to be attended as well as those that can skip a transformer layer. We show that ToSA can significantly reduce computation costs while maintaining accuracy on the ImageNet classification benchmark.
  arXiv  Detail & Related papers  (2024-06-13T05:17:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.