Visual Access Boundaries in Vision-Language Model Reasoning

• URL: http://arxiv.org/abs/2607.12815v2
• Date: Sat, 18 Jul 2026 02:07:01 GMT
• Title: Visual Access Boundaries in Vision-Language Model Reasoning
• Abstract summary: Chain-of-Thought (CoT) prompting is widely used as a test-time scaling strategy for Vision-Language Models (VLMs)<n>We show that CoT does not primarily improve performance by prolonging direct image-token access throughout the reasoning trace.<n>We also show that CoT gains are constrained by perceptual readout.
• Score: 23.49046016297019
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Chain-of-Thought (CoT) prompting is widely used as a test-time scaling strategy for Vision-Language Models (VLMs), but it remains unclear what is extended when VLMs generate longer reasoning traces. We ask whether CoT requires continued access to image tokens, or whether it mainly operates over visual information already made available earlier in the forward pass. We introduce Visual Access Sweep, a causal intervention that masks attention from generated-token queries to image-token keys along layer depth and generation time, and define the Visual Access Boundary (VAB) as the minimal access region that preserves task accuracy. Across six model configurations from Qwen2.5-VL and InternVL3, both no-CoT direct answering and CoT prompting exhibit finite VABs. In Qwen2.5-VL-32B and InternVL3 at 14B and 38B scales, when CoT is evaluated against the no-CoT full-access target, its VAB layer differs from the no-CoT boundary by at most two layers, despite substantially longer generations. This suggests that CoT does not primarily improve performance by prolonging direct image-token access throughout the reasoning trace, but by extending language-side computation over image-derived hidden-state information. We further show that CoT gains are constrained by perceptual readout. CoT helps when the queried visual attribute can be reliably read out by the model, but not when that readout is unreliable. A symbolic-attribute oracle shows that CoT can improve counting once ground-truth attributes are supplied as text, while a single-object probe-vs-decode check shows that hard attributes can be linearly recoverable from hidden states yet difficult for the model itself to output. Together, these analyses place the bottleneck at readout rather than counting.

Related papers

• GLaQ: Grounding Latent Queries in Visual Evidence for Multimodal Reasoning [27.232897918541067]
  Chain-of-thought reasoning has substantially improved the problem-solving capabilities of multimodal large language models.<n>Fine-grained visual evidence, however, remains difficult to preserve and reuse across text-based reasoning steps.<n>We introduce GLaQ, a grounded latent-query framework that replaces sequential latent rollout with a fixed set of context-conditioned queries.
  arXiv  Detail & Related papers  (2026-08-16T04:01:08Z)
• LAST: The Last Query Token Guides Visual Token Pruning for Edge-Cloud Collaborative MLLM Inference [63.932718076408584]
  LAST is a training-free framework for query-dependent visual token pruning in edge-cloud collaborative MLLM inference.<n>It preserves 95.4% of the full-token accuracy while retaining only 12.5% of the visual tokens, with low edge-side selection overhead and reduced cloud-side computation.
  arXiv  Detail & Related papers  (2026-07-30T09:59:25Z)
• Reading Between the Dots: Decoding Hidden Computation across Filler Tokens [7.203846680259872]
  LLMs can perform multi-step reasoning over content-free filler tokens like dots or counting sequences, producing correct answers with no visible chain-of-thought (CoT)<n>This is a limit case for behavioral oversight, where surface tokens carry no information about the underlying reasoning.<n>We introduce an unsupervised decoding pipeline that takes only hidden states as input and recovers intermediate values with 80-95% accuracy.
  arXiv  Detail & Related papers  (2026-07-03T17:18:34Z)
• Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference [13.079143204091705]
  We study visual-token from an answer-observable perspective and find that late visual-token updates can remain large while having little effect on answer-token representations.<n>We propose an operator-level visual-token skipping framework that preserves the full visual-token sequence while selectively bypassing redundant attention, FFN, or both.
  arXiv  Detail & Related papers  (2026-06-30T16:08:29Z)
• ATLAS: Agentic or Latent Visual Reasoning? One Word is Enough for Both [55.182037225013836]
  ATLAS is a framework in which a single discrete 'word', termed as a functional token, serves both as an agentic operation and a latent visual reasoning unit.<n> ATLAS achieves superior performance on challenging benchmarks while maintaining clear interpretability.
  arXiv  Detail & Related papers  (2026-05-14T17:59:55Z)
• ResPrune: Text-Conditioned Subspace Reconstruction for Visual Token Pruning in Large Vision-Language Models [59.94664910790462]
  ResPrune is a training-free visual token pruning framework for large vision-language models.<n>It selects a compact yet informative subset of visual tokens.<n>It achieves effective reductions in computation, memory consumption, and inference latency.
  arXiv  Detail & Related papers  (2026-03-22T07:44:45Z)
• All You Need Are Random Visual Tokens? Demystifying Token Pruning in VLLMs [43.80391827200227]
  In deep layers, existing training-free pruning methods perform no better than random pruning.<n>Visual tokens progressively lose their salience with increasing network depth.<n>We show that simple random pruning in deep layers efficiently balances performance and efficiency.
  arXiv  Detail & Related papers  (2025-12-08T14:16:01Z)
• SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference [49.84148668264725]
  We present SparseVILA, a new paradigm for efficient VLM inference that decouples visual sparsity across the prefilling and decoding stages.<n>Built on an AWQ-optimized inference pipeline, SparseVILA achieves up to 4.0 times faster prefilling, 2.5 times faster decoding, and an overall 2.6 times end-to-end speedup on long-context video tasks.
  arXiv  Detail & Related papers  (2025-10-20T17:35:47Z)
• Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimization [69.29207684569695]
  Chain-of-thought (CoT) reasoning greatly improves the interpretability and problem-solving abilities of multimodal large language models (MLLMs)<n>Existing approaches are focused on text CoT, limiting their ability to leverage visual cues.<n>In this paper, we introduce Unsupervised Visual CoT (UV-CoT), a novel framework for image-level CoT reasoning via preference optimization.
  arXiv  Detail & Related papers  (2025-04-25T14:48:18Z)
• QuoTA: Query-oriented Token Assignment via CoT Query Decouple for Long Video Comprehension [86.0749609778104]
  We propose QuoTA, an ante-hoc training-free modular that extends existing large video-language models.<n>QuoTA strategically allocates frame-level importance scores based on query relevance.<n>We decouple the query through Chain-of-Thoughts reasoning to facilitate more precise LVLM-based frame importance scoring.
  arXiv  Detail & Related papers  (2025-03-11T17:59:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.