Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models

• URL: http://arxiv.org/abs/2605.20950v1
• Date: Wed, 20 May 2026 09:37:53 GMT
• Title: Focus-then-Context: Subject-Centric Progressive Visual Token Reduction for Vision-Language Models
• Abstract summary: SPpruner is a subject-centric progressive reduction paradigm.<n>It emulates the textitFocus-then-Context mechanism of the human visual perception system.<n>It can excavate the comprehensive visual subject spectrum to ensure a high-fidelity representation of visual input.
• Score: 16.39553879497114
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language Models (VLMs) face a bottleneck of prohibitive computational costs arising from massive visual token sequences during inference. Existing vision token reduction methods alleviate this burden, but they unintentionally preserve the isolated visual subject strictly aligned with the user's query, which fails to substantially explore salient subjects and their contextual relationships. In this paper, we propose SPpruner, a subject-centric progressive reduction paradigm that emulates the \textit{Focus-then-Context} mechanism of the human visual perception system. Specifically, we first construct a focus identification module to explicitly model the interplay between visual saliency and semantic relevance. Herein, it can excavate the comprehensive visual subject spectrum to ensure a high-fidelity representation of visual input. Subsequently, a context-aware structural scanning module is developed to aggregate contextual cues from neighboring regions. As such, it can effectively restore global relational dependencies to uphold the structural integrity of the preserved subjects. Extensive experiments demonstrate that our paradigm consistently outperforms SOTA methods, achieving up to 2.53 times speedup with only 22.2% of visual tokens retained in Qwen2.5-VL and a 67% FLOPs reduction on LLaVA with a negligible 0.6% accuracy drop.

Related papers

• When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semantics [14.506581918176478]
  We introduce Structure-to-Semantics (STS), a novel two-stage visual token pruning framework.<n>The first stage employs a repulsion-based sampling mechanism to maximize spatial and structural diversity.<n>The second stage leverages instruction-aware cross-attention to precisely filter out prompt-irrelevant tokens.
  arXiv  Detail & Related papers  (2026-06-02T12:36:24Z)
• Molecular Identifier Visual Prompt and Verifiable Reinforcement Learning for Chemical Reaction Diagram Parsing [52.825281124618535]
  Reaction diagram parsing (RxnDP) is critical for extracting chemical synthesis information from literature.<n>Recent Vision-Language Models (VLMs) have emerged as a promising paradigm to automate this complex visual reasoning task.<n>This work enhances VLM-based RxnDP from two complementary perspectives: prompting representation and learning paradigms.
  arXiv  Detail & Related papers  (2026-03-16T09:17:05Z)
• One Token, Two Fates: A Unified Framework via Vision Token Manipulation Against MLLMs Hallucination [46.864482139261675]
  Current training-free methods tackle MLLM hallucination with separate strategies.<n>We propose a unified framework by focusing on the core asset: the vision token.<n>By harmonizing these two roles, our framework effectively restores the vision-language balance.
  arXiv  Detail & Related papers  (2026-03-11T03:19:46Z)
• Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning [82.39668822222386]
  Vision token pruning has proven to be an effective acceleration technique for the efficient Vision Language Model (VLM)<n>We propose $textNwa$, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity.<n>Experiments demonstrate that $textNwa$ achieves SOTA performance on multiple VQA benchmarks (from 94% to 95%) and yields substantial improvements on visual grounding tasks (from 7% to 47%)
  arXiv  Detail & Related papers  (2026-02-03T00:51:03Z)
• Thinking with Deltas: Incentivizing Reinforcement Learning via Differential Visual Reasoning Policy [75.66913260900726]
  Reinforcement Learning with Verifiable Rewards has significantly advanced reasoning capabilities in Large Language Models.<n>Existing paradigms, driven by text-centric outcome rewards, encourage models to bypass visual perception.<n>We propose textbfThinking with Deltas, a framework driven by a textbfDifferential Visual Reasoning Policy.
  arXiv  Detail & Related papers  (2026-01-11T08:25:34Z)
• Stable Language Guidance for Vision-Language-Action Models [62.80963701282789]
  Residual Semantic Steering is a probabilistic framework that disentangles physical affordance from semantic execution.<n> RSS achieves state-of-the-art robustness, maintaining performance even under adversarial linguistic perturbations.
  arXiv  Detail & Related papers  (2026-01-07T16:16:10Z)
• ViPER: Empowering the Self-Evolution of Visual Perception Abilities in Vision-Language Model [61.29164681694533]
  ViPER is a self-bootstrapping framework specifically designed to enable iterative evolution through self-critiquing and self-prediction.<n>Qwen-Viper consistently demonstrates superior performance across different vision-language scenarios while maintaining generalizability.
  arXiv  Detail & Related papers  (2025-10-28T10:42:57Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Rethinking Visual Token Reduction in LVLMs under Cross-modal Misalignment [38.04426918886084]
  Vision-Language Models (LVLMs) encode visual inputs as dense sequences of patch-level tokens to capture fine-grained semantics.<n>Previous efforts have explored visual token reduction either prior to or within the large language models (LLMs)<n>We introduce VisionDrop, a training-free, visual-only pruning framework that selects informative visual tokens based on intra-modal (visual-to-visual) attention.
  arXiv  Detail & Related papers  (2025-06-27T14:55:40Z)
• Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs [9.406760867809124]
  This paper introduces Visual Input Structure for Enhanced Reasoning (VISER)<n>VISER is a simple, effective method that augments visual inputs with low-level spatial structures.<n>We empirically demonstrate substantial performance improvements across core visual reasoning tasks.
  arXiv  Detail & Related papers  (2025-06-27T11:44:40Z)
• Seeing is Believing? Mitigating OCR Hallucinations in Multimodal Large Language Models [24.363156120809546]
  We propose KIE-HVQA, the first benchmark dedicated to evaluating OCR hallucination in degraded document understanding.<n>This dataset includes test samples spanning identity cards and invoices, with simulated real-world degradations for OCR reliability.<n>Experiments on Qwen2.5-VL demonstrate that our 7B- parameter model achieves a 22% absolute improvement in hallucination-free accuracy over GPT-4o.
  arXiv  Detail & Related papers  (2025-06-25T06:44:07Z)
• "Principal Components" Enable A New Language of Images [79.45806370905775]
  We introduce a novel visual tokenization framework that embeds a provable PCA-like structure into the latent token space.<n>Our approach achieves state-of-the-art reconstruction performance and enables better interpretability to align with the human vision system.
  arXiv  Detail & Related papers  (2025-03-11T17:59:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.