GOTS: Greedy Orthogonal Token Selection for High-Resolution Vision-Language Models

• URL: http://arxiv.org/abs/2607.23913v1
• Date: Mon, 27 Jul 2026 01:06:20 GMT
• Title: GOTS: Greedy Orthogonal Token Selection for High-Resolution Vision-Language Models
• Abstract summary: Greedy Orthogonal Token Selection (GOTS) is a training-free and query-agnostic method for visual token reduction.<n> GOTS achieves higher average performance retention than the strongest evaluated baselines.<n>It reduces model-side time-to-first-token after accounting for OCR selection overhead.
• Score: 9.490341327220987
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern vision-language models (VLMs) increasingly rely on dynamic or high-resolution visual encoding, producing thousands of visual tokens that substantially increase downstream language-model inference cost. Existing token-reduction methods assess token utility through token-wise importance, query relevance, coverage, pairwise diversity, or subset-level objectives. Our key insight is to view visual token reduction through selected-span complementarity: instead of scoring a token in isolation or through pairwise relations, we assess how much of its feature is orthogonal to the span of the already retained subset. Based on this perspective, we propose Greedy Orthogonal Token Selection (GOTS), a training-free and query-agnostic method. At each step, GOTS selects the token with the largest residual energy orthogonal to the current retained span. This rule exactly maximizes the one-step augmented Gram determinant among candidate additions, giving each greedy step a precise local geometric guarantee for subset expansion. Across five high-resolution VLM backbones from the Qwen-VL and InternVL families and eleven diverse benchmarks, GOTS achieves higher average performance retention than the strongest evaluated baselines, and a controlled OCRBench study shows that it reduces model-side time-to-first-token after accounting for selection overhead. Code is available at https://github.com/newLLing/GOTS.

Related papers

• StackTok: Accelerating VLMs Inference with Budget-Adaptive Visual Token Selection [17.809800994999225]
  StackTok is a training-free selector that treats query relevance as the objective and visual coverage as budget-calibrated support.<n>It ranks first among training-free selectors in every tested model--budget setting.<n>On high-resolution LLaVA-NeXT-7B, it retains 95.26% of full-token performance with only 160 of 2,880 (5.6%) visual tokens.
  arXiv  Detail & Related papers  (2026-09-15T08:35:57Z)
• Stepwise Token Selection for Efficient Multimodal Large Language Models [0.7233065479782755]
  We introduce a pointer-style selection mechanism that iteratively chooses informative tokens, conditioning each decision on previously selected ones, and dynamically determines when to stop.<n>Under aggressive pruning that removes 88.9% of visual tokens, our method preserves 94.6% of the original accuracy while achieving a 1.88x speed-up in prefill latency.
  arXiv  Detail & Related papers  (2026-06-14T23:50:08Z)
• [CLS] is Not Enough: Multi-Label Recognition via Patch-Level Inference and Adaptive Aggregation [20.637119409165418]
  PIAA formulates prediction as Patch-level Inference followed by Adaptive Aggregation.<n>We introduce an adaptive aggregation module that consolidates patch-level scores into the final multi-label prediction.<n>Experiments show that our method achieves strong improvements with minimal extra computation.
  arXiv  Detail & Related papers  (2026-05-25T13:19:12Z)
• Unlearning What Matters: Token-Level Attribution for Precise Language Model Unlearning [5.454773103061359]
  TokenUnlearn is a token-level attribution framework that identifies and selectively targets critical tokens.<n>Our approach combines knowledge-aware signals via masking, and entropy-aware signals to yield importance scores for precise token selection.
  arXiv  Detail & Related papers  (2026-05-01T02:59:03Z)
• Continuous Autoregressive Language Models [56.49239051750678]
  We introduce Continuous Autoregressive Language Models (CALM)<n>CALM uses a high-fidelity autoencoder to compress a chunk of K tokens into a single continuous vector.<n>We develop a comprehensive likelihood-free framework that enables robust training, evaluation, and controllable sampling.
  arXiv  Detail & Related papers  (2025-10-31T17:58:11Z)
• SCOPE: Saliency-Coverage Oriented Token Pruning for Efficient Multimodel LLMs [59.415473779171315]
  We propose a novel visual token pruning strategy called textbfSaliency-textbfCoverage textbfOriented token textbfPruning for textbfEfficient MLLMs.
  arXiv  Detail & Related papers  (2025-10-28T09:29:37Z)
• ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning [51.133569963553576]
  ssToken is a Self-modulated and Semantic-aware Token Selection approach.<n>We show that both self-modulated selection and semantic-aware selection alone outperform full-data fine-tuning.
  arXiv  Detail & Related papers  (2025-10-21T03:21:04Z)
• ZSPAPrune: Zero-Shot Prompt-Aware Token Pruning for Vision-Language Models [7.7352936204066]
  We propose a novel, zero-shot method to model visual token pruning as a balance between task relevance and information diversity.<n>Our method achieves performance that matches or surpasses the state-of-the-art with only minimal accuracy loss.<n>These gains are accompanied by significant reductions in GPU memory footprint and inference latency.
  arXiv  Detail & Related papers  (2025-10-20T06:18:47Z)
• IGD: Token Decisiveness Modeling via Information Gain in LLMs for Personalized Recommendation [79.22388408461458]
  We introduce an Information Gain-based Decisiveness-aware Token handling (IGD) strategy that integrates token decisiveness into both tuning and decoding.<n>IGD consistently improves recommendation accuracy, achieving significant gains on widely used ranking metrics compared to strong baselines.
  arXiv  Detail & Related papers  (2025-06-16T08:28:19Z)
• Reinforcing Video Reasoning with Focused Thinking [65.85683941058916]
  We propose TW-GRPO, a novel framework that enhances visual reasoning with focused thinking and dense reward granularity.<n>Specifically, we employ a token weighting mechanism that prioritizes tokens with high informational density.<n>We also reformulate RL training by shifting from single-choice to multi-choice QA tasks.
  arXiv  Detail & Related papers  (2025-05-30T15:42:19Z)
• Vector Quantized Wasserstein Auto-Encoder [57.29764749855623]
  We study learning deep discrete representations from the generative viewpoint. We endow discrete distributions over sequences of codewords and learn a deterministic decoder that transports the distribution over the sequences of codewords to the data distribution. We develop further theories to connect it with the clustering viewpoint of WS distance, allowing us to have a better and more controllable clustering solution.
  arXiv  Detail & Related papers  (2023-02-12T13:51:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.