Centering before Pruning: Lightweight Geometry Correction for Diversity-Based Visual Token Pruning in LVLMs

• URL: http://arxiv.org/abs/2608.30263v1
• Date: Mon, 31 Aug 2026 05:13:29 GMT
• Title: Centering before Pruning: Lightweight Geometry Correction for Diversity-Based Visual Token Pruning in LVLMs
• Abstract summary: We find that similarities between raw visual tokens are strongly concentrated in the positive range, limiting their ability to distinguish non-redundant tokens.<n>We propose the textbfCentered Geometry textbfPruner (Cen-Prune), which measures subset diversity using centered cosine similarity.<n>Cen-Prune provides robust improvements in overall performance across existing diversity-based pruners.
• Score: 7.371189496638082
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large vision-language models (LVLMs) incur substantial inference costs due to their long and highly redundant visual-token sequences. Diversity-based pruning mitigates this cost by selecting token subsets based on pairwise cosine similarity. We find, however, that similarities between raw visual tokens are strongly concentrated in the positive range, limiting their ability to distinguish non-redundant tokens. A natural way to improve this resolution is to center token features before computing cosine similarity. Centering indeed reveals a substantially richer pairwise structure, yet unexpectedly degrades pruning performance when used alone. We show that this apparent contradiction arises because the raw geometry does more than represent pairwise diversity: it also implicitly favors globally distinctive tokens, which tend to contain semantically informative content. Centering better resolves subset diversity but loses this useful token-wise preference, revealing that diversity and distinctiveness are entangled in the raw geometry. Based on this analysis, we propose the \textbf{Cen}tered Geometry \textbf{Prune}r (Cen-Prune), which measures subset diversity using centered cosine similarity while retaining raw-space distinctiveness as a complementary token-wise preference. This lightweight, plug-and-play correction leaves the underlying selection mechanism unchanged and incurs negligible computational overhead. Extensive experiments across multiple image- and video-understanding benchmarks and LVLM architectures demonstrate that Cen-Prune provides robust improvements in overall performance across existing diversity-based pruners.

Related papers

• ECOKV: Geometry-Aware KV Cache Eviction via Complementary Diversity Metrics [15.835203260282746]
  multimodal Large Language Models (MLLMs) excel in diverse tasks, but their scalability remains limited by the memory and computational overhead of KV cache storage.<n>Recent KV cache eviction approaches incorporate a cosine similarity-based diversity metric with importance metrics to selectively retain key-value pairs.<n>We propose a geometry-aware composite metric that jointly leverages Euclidean distance and cosine similarity to capture token diversity from complementary perspectives.
  arXiv  Detail & Related papers  (2026-09-06T15:13:48Z)
• Moving Beyond Diversity: Visual Token Pruning as Subspace Reconstruction for Efficient VLMs [7.371189496638082]
  We introduce SPARE, a subspace reconstruction method that reformulates token pruning as a column subset selection problem.<n>We show that SPARE consistently achieves state-of-the-art performance, with strong gains on tasks.
  arXiv  Detail & Related papers  (2026-06-17T04:45:10Z)
• MaskAlign: Token-Subset Representation Alignment for Efficient Diffusion Training [53.777757218665386]
  MaskAlign is a token-subset representation alignment method that applies alignment to randomly sampled token subsets during training.<n>We show that MaskAlign reduces the dependence of representation alignment on the complete token set and encourages alignment behavior that is more stable under token-subset perturbations.
  arXiv  Detail & Related papers  (2026-06-07T19:11:22Z)
• Exploiting Discriminative Codebook Prior for Autoregressive Image Generation [54.14166700058777]
  token-based autoregressive image generation systems first tokenize images into sequences of token indices with a codebook, and then model these sequences in an autoregressive paradigm.<n>While autoregressive generative models are trained only on index values, the prior encoded in the codebook, which contains rich token similarity information, is not exploited.<n>Recent studies have attempted to incorporate this prior by performing naive k-means clustering on the tokens, helping to facilitate the training of generative models with a reduced codebook.<n>We propose the Discriminative Codebook Prior Extractor (DCPE) as an alternative to k-means
  arXiv  Detail & Related papers  (2025-08-14T15:00:00Z)
• Training-Free Tokenizer Transplantation via Orthogonal Matching Pursuit [45.18582668677648]
  We present a training-free method to transplant tokenizers in large language models.<n>We approximate each out-of-vocabulary token as a sparse linear combination of shared tokens.<n>We show that OMP achieves best zero-shot preservation of the base model's performance.
  arXiv  Detail & Related papers  (2025-06-07T00:51:27Z)
• Multi-Cue Adaptive Visual Token Pruning for Large Vision-Language Models [85.51753014478315]
  We introduce AdaptPrune, a novel plug-and-play training-free pruning method.<n>It builds on conventional attention-based pruning by integrating spatial distance and token similarity with an adaptive NMS approach.<n>Our approach ensures a comprehensive evaluation of token importance and substantially refines the pruning decisions.
  arXiv  Detail & Related papers  (2025-03-11T03:58:17Z)
• Unified Semantic and ID Representation Learning for Deep Recommenders [28.709935854073535]
  We propose a Unified Semantic and ID Representation Learning framework.<n>In our framework, ID tokens capture unique item attributes, while semantic tokens represent shared, transferable characteristics.<n>Our framework integrates cosine similarity in earlier layers and Euclidean distance in the final layer to optimize representation learning.
  arXiv  Detail & Related papers  (2025-02-23T07:17:28Z)
• The Double-Ellipsoid Geometry of CLIP [4.013156524547072]
  Contrastive Language-Image Pre-Training (CLIP) is highly instrumental in machine learning applications.<n>We show that text and image reside on linearly separable ellipsoid shells, not centered at the origin.<n>A new notion of conformity is introduced, which measures the average cosine similarity of an instance to any other instance.
  arXiv  Detail & Related papers  (2024-11-21T16:27:22Z)
• Token Fusion: Bridging the Gap between Token Pruning and Token Merging [71.84591084401458]
  Vision Transformers (ViTs) have emerged as powerful backbones in computer vision, outperforming many traditional CNNs. computational overhead, largely attributed to the self-attention mechanism, makes deployment on resource-constrained edge devices challenging. We introduce "Token Fusion" (ToFu), a method that amalgamates the benefits of both token pruning and token merging.
  arXiv  Detail & Related papers  (2023-12-02T04:29:19Z)
• Equivariant Similarity for Vision-Language Foundation Models [134.77524524140168]
  This study focuses on the multimodal similarity function that is not only the major training objective but also the core delivery to support downstream tasks. We propose EqSim, a regularization loss that can be efficiently calculated from any two matched training pairs. Compared to the existing evaluation sets, EqBen is the first to focus on "visual-minimal change"
  arXiv  Detail & Related papers  (2023-03-25T13:22:56Z)
• Deep Diversity-Enhanced Feature Representation of Hyperspectral Images [87.47202258194719]
  We rectify 3D convolution by modifying its topology to enhance the rank upper-bound. We also propose a novel diversity-aware regularization (DA-Reg) term that acts on the feature maps to maximize independence among elements. To demonstrate the superiority of the proposed Re$3$-ConvSet and DA-Reg, we apply them to various HS image processing and analysis tasks.
  arXiv  Detail & Related papers  (2023-01-15T16:19:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.