Balancing Saliency and Coverage: Semantic Prominence-Aware Budgeting for Visual Token Compression in VLMs

• URL: http://arxiv.org/abs/2603.14892v1
• Date: Mon, 16 Mar 2026 06:44:45 GMT
• Title: Balancing Saliency and Coverage: Semantic Prominence-Aware Budgeting for Visual Token Compression in VLMs
• Abstract summary: We propose a sample-adaptive visual token selection framework composed of semantic prominence-aware budget allocation and a two-stage selection pipeline.<n>On LLaVA-NeXT-7B, our approach reduces FLOPs by 88% and prefill latency by 22% while preserving 97.5% of the original accuracy.
• Score: 44.02979403592988
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Vision-Language Models (VLMs) achieve strong multimodal understanding capabilities by leveraging high-resolution visual inputs, but the resulting large number of visual tokens creates a major computational bottleneck. Recent work mitigates this issue through visual token compression, typically compressing tokens based on saliency, diversity, or a fixed combination of both. We observe that the distribution of semantic prominence varies substantially across samples, leading to different optimal trade-offs between local saliency preservation and global coverage. This observation suggests that applying a static compression strategy across all samples can be suboptimal. Motivated by this insight, we propose PromPrune, a sample-adaptive visual token selection framework composed of semantic prominence-aware budget allocation and a two-stage selection pipeline. Our method adaptively balances local saliency preservation and global coverage according to the semantic prominence distribution of each sample. By allocating token budgets between locally salient regions and globally diverse regions, our method maintains strong performance even under high compression ratios. On LLaVA-NeXT-7B, our approach reduces FLOPs by 88% and prefill latency by 22% while preserving 97.5% of the original accuracy.

Related papers

• CoverPrune: Coverage-Driven Token Pruning for 3D VLMs via Optimal Transport [50.2610873651284]
  Existing token pruning methods rely on diversity-based selection, discarding similar tokens to maximize dispersion.<n>In 3D environments, this approach frequently drops representative prototype tokens in favor of outliers.<n>We propose a paradigm shift for 3D VLM token pruning: from maximizing OT to preserving visual evidence coverage.
  arXiv  Detail & Related papers  (2026-08-13T13:29:15Z)
• Efficiency Follows Global-Local Decoupling [62.05489838893081]
  We introduce ConvNeur, a two-branch architecture in which a lightweight neural memory branch aggregates global context on a compact set of tokens.<n>A learned gate lets global cues modulate local features without entangling their objectives.<n>On standard classification, detection, and segmentation benchmarks, ConvNeur matches or surpasses comparable alternatives at similar or lower compute.
  arXiv  Detail & Related papers  (2026-03-20T02:20:16Z)
• LLMind: Bio-inspired Training-free Adaptive Visual Representations for Vision-Language Models [4.367067873280894]
  We present the first systematic analysis of bio-inspired visual representation methods.<n>We propose LLMind, a training-free framework that mimics foveated encoding and cortical magnification in human vision.<n>We show dramatic gains, with average improvements of +20% on VQAv2, +38% on Seed-Bench, and +37% on A-OKVQA compared to uniform sampling under tight pixel budgets.
  arXiv  Detail & Related papers  (2026-03-16T06:29:43Z)
• Prune Redundancy, Preserve Essence: Vision Token Compression in VLMs via Synergistic Importance-Diversity [54.95089105944234]
  Vision-language models (VLMs) face significant computational inefficiencies caused by excessive generation of visual tokens.<n>We propose PruneSID, a training-free Synergistic Importance-Diversity approach featuring a two-stage pipeline.<n>We demonstrate state-of-the-art performance, achieving 96.3% accuracy on LLaVA-1.5 with only 11.1% token retention, and 92.8% accuracy at extreme compression rates (5.6%) on LLaVA-NeXT.
  arXiv  Detail & Related papers  (2026-03-10T10:31:58Z)
• Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning [82.39668822222386]
  Vision token pruning has proven to be an effective acceleration technique for the efficient Vision Language Model (VLM)<n>We propose $textNwa$, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity.<n>Experiments demonstrate that $textNwa$ achieves SOTA performance on multiple VQA benchmarks (from 94% to 95%) and yields substantial improvements on visual grounding tasks (from 7% to 47%)
  arXiv  Detail & Related papers  (2026-02-03T00:51:03Z)
• Towards Reliable and Holistic Visual In-Context Learning Prompt Selection [82.23704441763651]
  Visual In-Context Learning (VICL) has emerged as a prominent approach for adapting visual foundation models to novel tasks.<n>VICL methods, such as Partial2Global and VPR, are grounded in the similarity-priority assumption that images more visually similar to a query image serve as better in-context examples.<n>This paper introduces an enhanced variant of Partial2Global designed for reliable and holistic selection of in-context examples in VICL.
  arXiv  Detail & Related papers  (2025-09-30T09:23:12Z)
• Beyond Hard and Soft: Hybrid Context Compression for Balancing Local and Global Information Retention [30.580674811560613]
  Large Language Models (LLMs) encounter significant challenges in long-sequence inference due to computational inefficiency and redundant processing.<n>Existing methods often rely on token importance to perform hard local compression or encode context into latent representations for soft global compression.<n>We propose HyCo$$, which integrates both global and local perspectives to guide context compression.
  arXiv  Detail & Related papers  (2025-05-21T17:26:11Z)
• Global Compression Commander: Plug-and-Play Inference Acceleration for High-Resolution Large Vision-Language Models [21.36437021964681]
  "Global Compression Commander" is a novel token compression framework for HR-LVLMs.<n>GlobalCom$2$ maintains over 90% performance while compressing 90% visual tokens, reducing FLOPs and peak memory to 9.1% and 60%.
  arXiv  Detail & Related papers  (2025-01-09T11:57:58Z)
• FedLAP-DP: Federated Learning by Sharing Differentially Private Loss Approximations [53.268801169075836]
  We propose FedLAP-DP, a novel privacy-preserving approach for federated learning. A formal privacy analysis demonstrates that FedLAP-DP incurs the same privacy costs as typical gradient-sharing schemes. Our approach presents a faster convergence speed compared to typical gradient-sharing methods.
  arXiv  Detail & Related papers  (2023-02-02T12:56:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.