StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection

• URL: http://arxiv.org/abs/2609.16841v1
• Date: Tue, 15 Sep 2026 08:35:57 GMT
• Title: StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection
• Abstract summary: StackTok is a training-free selector that treats query relevance as the objective and visual coverage as budget-calibrated support.<n>It ranks first among training-free selectors in every tested model--budget setting.<n>On high-resolution LLaVA-NeXT-7B, it retains 95.26% of full-token performance with only 160 of 2,880 (5.6%) visual tokens.
• Score: 17.809800994999225
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Increasing image resolution produces ever-longer visual-token sequences in vision-language models (VLMs), substantially raising their inference cost. To reduce this overhead without retraining, existing methods select compact token subsets that prioritize query relevance, visual coverage, or a fixed trade-off between them. The appropriate balance, however, varies across queries and token budgets: localized questions favor relevance, whereas holistic questions demand broader visual coverage. We introduce StackTok, a training-free selector that treats query relevance as the objective and visual coverage as budget-calibrated support. StackTok builds a size-indexed coverage reference from a coverage-only greedy sequence and adjusts its support target using query--vision affinity entropy. A reference-gated interleaved selection policy then switches between relevance- and coverage-oriented additions according to the current subset's support deficit. For high-resolution inputs, StackTok allocates one shared token budget across crops according to the combined marginal gain of locally nominated tokens. Evaluated with five VLMs over ten distinct image-understanding benchmarks, StackTok ranks first among training-free selectors in every tested model--budget setting. On high-resolution LLaVA-NeXT-7B, it retains 95.26% of full-token performance with only 160 of 2{,}880 (5.6%) visual tokens.

Related papers

• HAP: Head-Adaptive Visual Token Pruning via Cross-Modal Alignment [76.68875424389057]
  Existing methods score each visual token by averaging text-to-visual attention uniformly across all heads.<n> misaligned heads dominate the average, amplifying background tokens and drowning out fine-grained cues.<n>We propose PAQ (Prompt-Grounded Attention Quality), a metric quantifying how well each head aligns the prompt with image regions.
  arXiv  Detail & Related papers  (2026-08-24T23:55:36Z)
• LAST: The Last Query Token Guides Visual Token Pruning for Edge-Cloud Collaborative MLLM Inference [63.932718076408584]
  LAST is a training-free framework for query-dependent visual token pruning in edge-cloud collaborative MLLM inference.<n>It preserves 95.4% of the full-token accuracy while retaining only 12.5% of the visual tokens, with low edge-side selection overhead and reduced cloud-side computation.
  arXiv  Detail & Related papers  (2026-07-30T09:59:25Z)
• GOTS: Greedy Orthogonal Token Selection for High-Resolution Vision-Language Models [9.490341327220987]
  Greedy Orthogonal Token Selection (GOTS) is a training-free and query-agnostic method for visual token reduction.<n> GOTS achieves higher average performance retention than the strongest evaluated baselines.<n>It reduces model-side time-to-first-token after accounting for OCR selection overhead.
  arXiv  Detail & Related papers  (2026-07-27T01:06:20Z)
• Efficient Frame Selection for Long Videos at Test Time with Attention-Based MLLM Selectors [69.75895255395702]
  Cross-modal attention at validation-selected extraction layers in MLLMs already provides query-relevant frame evidence without requiring autoregressive generation.<n>A lightweight MLLM selector, even with only 2B parameters, can extract frame-level evidence by converting selected-layer attention into relevance scores through query-conditioned aggregation.<n>Under a 32-frame budget, our selector improves over uniform sampling by up to 6.4 points on Video-MME and outperforms prior training-based selectors under matched frame budgets.
  arXiv  Detail & Related papers  (2026-07-17T07:04:31Z)
• AnchorPrune: Relevance-Anchored Contextual Expansion for Visual Token Pruning [8.862195491555575]
  Large vision-language models incur substantial inference costs because high-resolution inputs introduce thousands of visual tokens.<n>We introduce AnchorPrune, a training-free framework that first constructs a protected relevance anchor and then expands it with visual context.<n>On LLaVA-NeXT-7B, AnchorPrune preserves 97.6% of full-token performance using only 160 of 2,880 visual tokens.
  arXiv  Detail & Related papers  (2026-07-08T06:07:06Z)
• Budget-Aware Routing for Long Clinical Text [8.474809035213118]
  Key challenge for large language models is token cost per query and overall deployment cost.<n>We study budgeted context selection, where a subset of document units is chosen under a strict token budget.<n>We propose textbfRCD, a monotone submodular objective that balances relevance, coverage, and diversity.
  arXiv  Detail & Related papers  (2026-05-01T01:34:53Z)
• Adaptive Greedy Frame Selection for Long Video Understanding [11.923839324117674]
  Large vision--language models (VLMs) are increasingly applied to long-video question answering.<n>Inference is often bottlenecked by the number of input frames and resulting visual tokens.<n>We propose a question-adaptive greedy frame selection method that jointly optimize query relevance and semantic representativeness under a fixed frame budget.
  arXiv  Detail & Related papers  (2026-03-20T17:55:32Z)
• VLM-Pruner: Buffering for Spatial Sparsity in an Efficient VLM Centrifugal Token Pruning Paradigm [18.604455802016233]
  Vision-language models (VLMs) excel at image understanding tasks, but the large number of visual tokens imposes significant computational costs.<n>We propose VLM-Pruner, a training-free token pruning algorithm that balances redundancy and spatial sparsity.<n>We show that VLM-Pruner consistently outperforms strong baselines across five VLMs with an 88.9% pruning rate, while delivering an end-to-end inference speedup.
  arXiv  Detail & Related papers  (2025-12-02T12:30:05Z)
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs [59.415473779171315]
  We propose a novel visual token pruning strategy called textbfSaliency-textbfCoverage textbfOriented token textbfPruning for textbfEfficient MLLMs.
  arXiv  Detail & Related papers  (2025-10-28T09:29:37Z)
• Reinforcing Video Reasoning with Focused Thinking [65.85683941058916]
  We propose TW-GRPO, a novel framework that enhances visual reasoning with focused thinking and dense reward granularity.<n>Specifically, we employ a token weighting mechanism that prioritizes tokens with high informational density.<n>We also reformulate RL training by shifting from single-choice to multi-choice QA tasks.
  arXiv  Detail & Related papers  (2025-05-30T15:42:19Z)
• QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long Video Comprehension [86.0749609778104]
  We propose QuoTA, an ante-hoc training-free modular that extends existing large video-language models.<n>QuoTA strategically allocates frame-level importance scores based on query relevance.<n>We decouple the query through Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame importance scoring.
  arXiv  Detail & Related papers  (2025-03-11T17:59:57Z)
• Activation-aware Probe-Query: Effective Key-Value Retrieval for Long-Context LLMs Inference [56.71209737306054]
  We propose textbfActQKV, a training-free, textbfActivation-aware approach that dynamically determines probe-textbfQuery and leverages it to retrieve the relevant textbfKV pairs for inference.<n>Experiments on the Long-Bench and $infty$ Benchmarks demonstrate its state-of-the-art performance with competitive inference quality and resource efficiency.
  arXiv  Detail & Related papers  (2025-02-19T08:50:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.