Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds

• URL: http://arxiv.org/abs/2608.21170v2
• Date: Mon, 24 Aug 2026 22:09:56 GMT
• Title: Is Visual Prompting All You Need? Studying VLM Spatial Reasoning under Progressive Visual Scaffolds
• Abstract summary: We study how the visual presentation of a task shapes model performance and failure modes when the underlying reasoning problem is unchanged.<n>We introduce lightweight input-side scaffolds that preserve the visual modality while making spatial structure more accessible.<n>We find that visual presentation is a central factor that determines whether VLM benchmarks measure grounded perception, downstream reasoning, or a mixture of both.
• Score: 12.841957617988148
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Vision-language models (VLMs) have advanced rapidly in multimodal reasoning, yet recent work shows that their failures often reflect an interaction between visual grounding and downstream reasoning. What remains less clear is how the visual presentation of a task shapes model performance and failure modes when the underlying reasoning problem is unchanged. We study this question in SPaRC, a benchmark for grid-based visual spatial planning, by introducing lightweight input-side scaffolds that preserve the visual modality while making spatial structure more accessible. Across multiple VLMs, these scaffolds improve task accuracy over the original visual setting by up to 34.0 percentage points and further complement GRPO-based training, yielding up to 4.6 additional accuracy points compared with near-zero gains on the original visual input. Analyses on both end-to-end task solving and object detection show that these gains are closely tied to reductions in grounding-related errors, while rule reasoning remains comparatively challenging. We find that visual presentation is a central factor that determines whether VLM benchmarks measure grounded perception, downstream reasoning, or a mixture of both.

Related papers

• The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoning [82.58503785830632]
  Vision-language models increasingly succeed on multimodal reasoning benchmarks, yet their visual evidence often becomes unstable once it enters the language stack.<n>We propose TRACE, a task-adaptive inference-time control framework with lightweight trained modules.<n>Across four open-weight VLM backbones and seven benchmarks, TRACE delivers large gains on grounding-sensitive settings.
  arXiv  Detail & Related papers  (2026-07-13T11:44:23Z)
• Imaginative Perception Tokens Enhance Spatial Reasoning in Multimodal Language Models [61.89854422423702]
  Vision language models (VLMs) excel at many tasks but struggle with spatial reasoning when critical information is not directly observable.<n>We introduce Imaginative Perception Tokens (IPT), intermediate perceptual representations that externalize what a VLM would perceive under alternative spatial configurations.<n>IPT supervision consistently improves spatial reasoning and often outperforms textual chain of thought training.
  arXiv  Detail & Related papers  (2026-06-02T17:59:17Z)
• Counting to Four is still a Chore for VLMs [3.017198998175514]
  Vision--language models (VLMs) have achieved impressive performance on complex multimodal reasoning tasks.<n>They still fail on simple grounding skills such as object counting.<n>We present an empirical study of VLM counting behavior through both behavioral and mechanistic analysis.
  arXiv  Detail & Related papers  (2026-04-11T05:23:19Z)
• ReViP: Reducing False Completion in Vision-Language-Action Models with Vision-Proprioception Rebalance [50.05984919728878]
  We present ReViP, a novel VLA framework with Vision-Proprioception Rebalance to enhance visual grounding and robustness under perturbations.<n>Specifically, we use an external VLM as a task-stage observer to extract real-time task-centric visual cues from visual observations.<n>To evaluate false completion, we propose the first False-Completion Benchmark Suite built on LIBERO with controlled settings such as Object-Drop.
  arXiv  Detail & Related papers  (2026-01-23T11:31:07Z)
• Unbiased Visual Reasoning with Controlled Visual Inputs [28.155426761798022]
  VISTA is a framework that decouples perception from reasoning via an explicit information bottleneck.<n>A frozen VLM sensor is restricted to short, objective perception queries.<n>A text-only LLM reasoner decomposes each question, plans queries, and aggregates visual facts in natural language.
  arXiv  Detail & Related papers  (2025-12-19T18:52:06Z)
• [De|Re]constructing VLMs' Reasoning in Counting [2.1856941852799134]
  We study the reasoning skills of seven state-of-the-art Vision-Language Models (VLMs) in the counting task under controlled experimental conditions.<n>A layer-wise analysis reveals that errors are due to incorrect mapping of the last-layer representation into the output space.<n>Our targeted training shows that fine-tuning just the output layer improves accuracy by up to 21%.
  arXiv  Detail & Related papers  (2025-10-22T13:08:47Z)
• Visual Structures Helps Visual Reasoning: Addressing the Binding Problem in VLMs [9.406760867809124]
  This paper introduces Visual Input Structure for Enhanced Reasoning (VISER)<n>VISER is a simple, effective method that augments visual inputs with low-level spatial structures.<n>We empirically demonstrate substantial performance improvements across core visual reasoning tasks.
  arXiv  Detail & Related papers  (2025-06-27T11:44:40Z)
• ViCrit: A Verifiable Reinforcement Learning Proxy Task for Visual Perception in VLMs [98.27348724529257]
  We introduce ViCrit (Visual Caption Hallucination Critic), an RL proxy task that trains VLMs to localize a subtle, synthetic visual hallucination injected into paragraphs of human-written image captions.<n>Models trained with the ViCrit Task exhibit substantial gains across a variety of vision-language models benchmarks.
  arXiv  Detail & Related papers  (2025-06-11T19:16:54Z)
• Can MLLMs Guide Me Home? A Benchmark Study on Fine-Grained Visual Reasoning from Transit Maps [56.76175383189738]
  We introduce ReasonMap, a benchmark designed to assess the fine-grained visual understanding and spatial reasoning abilities of MLLMs.<n>ReasonMap encompasses high-resolution transit maps from 30 cities across 13 countries and includes 1,008 question-answer pairs spanning two question types and three templates.<n> Comprehensive evaluations of 15 popular MLLMs, including both base and reasoning variants, reveal a counterintuitive pattern.
  arXiv  Detail & Related papers  (2025-05-24T12:33:52Z)
• Open Eyes, Then Reason: Fine-grained Visual Mathematical Understanding in MLLMs [62.875934732547435]
  Current large language models (MLLMs) often underperform on mathematical problem-solving tasks that require fine-grained visual understanding.<n>In this paper, we evaluate the visual grounding capabilities of state-of-the-art MLLMs and reveal a significant negative correlation between visual grounding accuracy and problem-solving performance.<n>We propose a novel approach, SVE-Math, featuring a geometric-grounded vision encoder and a feature router that dynamically adjusts the contribution of hierarchical visual feature maps.
  arXiv  Detail & Related papers  (2025-01-11T04:08:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.