From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankers

• URL: http://arxiv.org/abs/2609.00667v1
• Date: Tue, 01 Sep 2026 03:45:21 GMT
• Title: From Saliency to Discriminability: Rank-Preserving Visual Token Pruning for VLM Rerankers
• Abstract summary: Existing pruning methods retain tokens by attention saliency, yet we show that saliency is systematically misaligned with ranking contribution.<n>We propose RaDiCal, a training-free framework that uses normalized attention entropy to decide when saliency can be trusted.<n>It cuts FLOPs by 39--45% and delivers 1.28--1.45$times$ measured speedups across two VLM architectures without dataset-specific retuning.
• Score: 16.546400103214715
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large vision-language models used as listwise rerankers must jointly process visual tokens from tens of candidates per query, making token pruning essential for practical deployment. Existing pruning methods retain tokens by attention saliency, yet we show that saliency is systematically misaligned with ranking contribution: visually prominent tokens often capture order-neutral patterns shared across candidates. This mismatch is layer-dependent: saliency becomes informative only where attention is concentrated, and normalized attention entropy diagnoses the reliability shift (Pearson r=0.87). We propose RaDiCal (Rank-Discriminative Calibration), a training-free framework that uses normalized attention entropy to decide when saliency can be trusted, fusing it with an attention-free rank-discriminative prior and selecting pruning layers from the same trust landscape. Across three retrieval benchmarks and multiple VLM architectures, RaDiCal matches Dense MRR@10 on Flickr30K and surpasses it on MSCOCO at a 20% token budget, ranks first among all pruning methods on FashionIQ, and holds within 1.2 pp on Flickr30K and MSCOCO at 10% retention. It cuts FLOPs by 39--45% and delivers 1.28--1.45$\times$ measured speedups across two VLM architectures without dataset-specific retuning.

Related papers

• HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment [76.68875424389057]
  Existing methods score each visual token by averaging text-to-visual attention uniformly across all heads.<n> misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues.<n>We propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions.
  arXiv  Detail & Related papers  (2026-08-24T23:55:36Z)
• Calibrate Before Reason: Robust Visual Token Reduction against Semantic Drift in VLMs [26.936352183867854]
  CaRe consists of two mutually complementary modules: 1) Perturbation-Robust Anchoring, which identifies anchors with stable model-side influence under multi-directional perturbations; 2) Confidence-Gated Token, which extracts reliable calibration signals from unselected tokens and injects them into anchors.<n>While pruning 94.4% of visual tokens, our method retains 96.4% of the original full-token performance, delivering up to 2.30 times faster end-to-end inference speed relative to unpruned vanilla models.
  arXiv  Detail & Related papers  (2026-07-30T05:36:29Z)
• TOPS: First-Principles Visual Token Pruning via Constructing Token Optimal Preservation Sets for Efficient MLLM Inference [54.41143483594129]
  Multimodal large language models (MLLMs) have achieved strong multimodal reasoning capabilities, but their efficiency is limited by the large number of visual tokens.<n>Visual token pruning offers a natural solution, yet existing methods are imperfect.<n>We propose TOPS, a training-free and model-agnostic pruning module that can be applied to various MLLMs.
  arXiv  Detail & Related papers  (2026-06-25T15:29:37Z)
• OccamToken: Efficient VLM Inference with Training-Free and Budget-Adaptive Token Pruning [24.164883144694656]
  OccamToken is a training-free framework that replaces absolute token ranking with register-anchored relative evidence testing.<n>We show that OccamToken consistently improves the accuracy-efficiency trade-off without additional training.
  arXiv  Detail & Related papers  (2026-05-28T09:20:47Z)
• Decoupled Similarity for Task-Aware Token Pruning in Large Vision-Language Models [73.07332521794093]
  Token pruning has emerged as an effective approach to reduce the substantial computational overhead of Large Vision-Language Models.<n>We propose DeSAP, a novel Decoupled Similarity-Aware Pruning method for precise, task-aware token pruning within the visual encoder.<n>DeSAP consistently outperforms SOTA methods in both accuracy and efficiency.
  arXiv  Detail & Related papers  (2026-04-13T09:44:52Z)
• When Does Content-Based Routing Work? Representation Requirements for Selective Attention in Hybrid Sequence Models [0.0]
  We identify a routing paradox in hybrid recurrent-attention architectures.<n>We show that content-based routing requires exactly the pairwise computation that routing is designed to avoid.
  arXiv  Detail & Related papers  (2026-03-22T01:04:57Z)
• Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning [82.39668822222386]
  Vision token pruning has proven to be an effective acceleration technique for the efficient Vision Language Model (VLM)<n>We propose $textNwa$, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity.<n>Experiments demonstrate that $textNwa$ achieves SOTA performance on multiple VQA benchmarks (from 94% to 95%) and yields substantial improvements on visual grounding tasks (from 7% to 47%)
  arXiv  Detail & Related papers  (2026-02-03T00:51:03Z)
• vAttention: Verified Sparse Attention [100.98210818821688]
  vAttention is a practical sparse attention mechanism with user-specified $(epsilon, delta)$ guarantees on approximation accuracy (thus, verified)<n>We show that vAttention significantly improves the quality of sparse attention across datasets.<n>It can be deployed in reasoning scenarios to achieve fast decoding without compromising model quality.
  arXiv  Detail & Related papers  (2025-10-07T08:46:08Z)
• HiPrune: Training-Free Visual Token Pruning via Hierarchical Attention in Vision-Language Models [6.306822764683807]
  HiPrune is a training-free and model-agnostic token Pruning framework for vision encoders.<n>It exploits the Hierarchical attention structure within vision encoders.<n>It preserves up to 99.3% task accuracy with only 33.3% tokens, and maintaining 99.5% accuracy with just 11.1% tokens.
  arXiv  Detail & Related papers  (2025-08-01T11:48:11Z)
• CLIPure: Purification in Latent Space via CLIP for Adversarially Robust Zero-Shot Classification [65.46685389276443]
  We ground our work on CLIP, a vision-language pre-trained encoder model that can perform zero-shot classification by matching an image with text prompts.<n>We then formulate purification risk as the KL divergence between the joint distributions purification process.<n>We propose two variants for our CLIPure approach: CLI-Diff which models the likelihood of images' latent vectors, and CLIPure-Cos which models the likelihood with the cosine similarity between the embeddings of an image and a photo of a.''
  arXiv  Detail & Related papers  (2025-02-25T13:09:34Z)
• S2-Attention: Hardware-Aware Context Sharding Among Attention Heads [49.1454481007861]
  Sparse attention selectively attends to a subset of tokens in the context.<n>It remains unclear whether sparse attention can maintain the model's quality at a scale of today's large language models.<n>This paper presents Sparsely-Sharded(S2) Attention, a Triton library that provides kernel optimization for sparse attention customizable at both per-head and per-context-range levels.
  arXiv  Detail & Related papers  (2024-07-25T00:27:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.