When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics

• URL: http://arxiv.org/abs/2606.03569v1
• Date: Tue, 02 Jun 2026 12:36:24 GMT
• Title: When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics
• Abstract summary: We introduce Structure-to-Semantics (STS), a novel two-stage visual token pruning framework.<n>The first stage employs a repulsion-based sampling mechanism to maximize spatial and structural diversity.<n>The second stage leverages instruction-aware cross-attention to precisely filter out prompt-irrelevant tokens.
• Score: 14.506581918176478
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) have demonstrated remarkable capabilities but suffer from significant computational overhead during inference. While visual token pruning offers a promising solution, existing methods predominantly rely on initial attention scores. This single-metric paradigm presents a critical flaw: high attention scores inherently collapse onto semantically similar regions, thereby severely reducing feature diversity and discarding vital contextual details. To address this, we introduce Structure-to-Semantics (STS), a novel two-stage visual token pruning framework that explicitly decouples the pruning process. The first stage employs a repulsion-based sampling mechanism to maximize spatial and structural diversity. The second stage leverages instruction-aware cross-attention to precisely filter out prompt-irrelevant tokens. This two-stage synergy constitutes the core of STS, first ensuring geometric coverage and then refining the retained tokens according to semantic relevance. Extensive evaluations demonstrate that STS mitigates the redundancy caused by attention-based selection, improving both structural diversity and fine-grained task alignment of the preserved visual tokens.

Related papers

• SSC-Priors: Exploring Semantic and Visibility Priors to Boost Lidar Semantic Scene Completion [61.665449818739155]
  This paper investigates easy strategies to boost the performance of existing networks for lidar semantic scene completion (SSC)<n>We use semantic pseudo-labels and sensor visibility information to improve existing approaches.<n>We show that endowing input point clouds with semantic pseudo-labels from off-the-shelf segmenters significantly improves the performance of existing SSC models.
  arXiv  Detail & Related papers  (2026-09-15T16:37:46Z)
• Disentangling Semantic Attention from Structural Bias in the Attention Manifold [61.68044165974505]
  Multimodal Large Language Models (MLLMs) exhibit disproportionate attention toward certain semantically uninformative visual tokens.<n>We introduce Saliency-guided Purification and Adaptive Redistribution (SPAR), a training-free, plug-and-play intervention.
  arXiv  Detail & Related papers  (2026-07-27T05:30:19Z)
• Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models [16.39553879497114]
  SPpruner is a subject-centric progressive reduction paradigm.<n>It emulates the textitFocus-then-Context mechanism of the human visual perception system.<n>It can excavate the comprehensive visual subject spectrum to ensure a high-fidelity representation of visual input.
  arXiv  Detail & Related papers  (2026-05-20T09:37:53Z)
• Evading Visual Aphasia: Contrastive Adaptive Semantic Token Pruning for Vision-Language Models [52.78477729846771]
  We introduce COAST (COntrastive Adaptive Semantic Token Pruning), a training-free pruning framework that casts compression as adaptive semantic routing.<n> COAST consistently outperforms strong pruning baselines across token budgets and generalizes across multiple LVLM families.
  arXiv  Detail & Related papers  (2026-05-10T09:07:04Z)
• Beyond Static Visual Tokens: Structured Sequential Visual Chain-of-Thought Reasoning [8.10005273393957]
  We propose Structural Sequential Visual CoT SSV-CoT.<n>First, a question-relevant saliency map identifies and organizes key visual regions, explicitly modeling the spatial distribution of visual importance.<n>Second, reasoning is performed following this discriminative order, inducing a curriculum-like semantic progression from primary to secondary cues.
  arXiv  Detail & Related papers  (2026-03-21T20:13:15Z)
• Unify the Views: View-Consistent Prototype Learning for Few-Shot Segmentation [23.546777614096424]
  Few-shot segmentation (FSS) has gained significant attention for its ability to generalize to novel classes with limited supervision.<n>This paper tackles these challenges by introducing VINE, a unified framework that models structural consistency and discrimination to refine class-specific prototypes.
  arXiv  Detail & Related papers  (2026-03-06T06:36:42Z)
• Punctuation-aware Hybrid Trainable Sparse Attention for Large Language Models [44.28116882776357]
  We present textbfPunctuation-aware textbfHybrid textbfSparse textbfAttention textbf(PHSA), a trainable sparse attention framework that leverages punctuation tokens as semantic boundary anchors.<n>Specifically, we design a dual-branch aggregation mechanism that fuses global semantic representations with punctuation-enhanced boundary features, preserving the core semantic structure while introducing almost no additional computational overhead.
  arXiv  Detail & Related papers  (2026-01-06T08:47:16Z)
• ToDRE: Visual Token Pruning via Diversity and Task Awareness for Efficient Large Vision-Language Models [59.47738955960352]
  ToDRE is a two-stage and training-free token compression framework.<n>It achieves superior performance by pruning tokens based on token Diversity and token-task RElevance.
  arXiv  Detail & Related papers  (2025-05-24T15:47:49Z)
• "Principal Components" Enable A New Language of Images [79.45806370905775]
  We introduce a novel visual tokenization framework that embeds a provable PCA-like structure into the latent token space.<n>Our approach achieves state-of-the-art reconstruction performance and enables better interpretability to align with the human vision system.
  arXiv  Detail & Related papers  (2025-03-11T17:59:41Z)
• Semantics Meets Temporal Correspondence: Self-supervised Object-centric Learning in Videos [63.94040814459116]
  Self-supervised methods have shown remarkable progress in learning high-level semantics and low-level temporal correspondence. We propose a novel semantic-aware masked slot attention on top of the fused semantic features and correspondence maps. We adopt semantic- and instance-level temporal consistency as self-supervision to encourage temporally coherent object-centric representations.
  arXiv  Detail & Related papers  (2023-08-19T09:12:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.