Thinking with Visual Grounding

• URL: http://arxiv.org/abs/2606.16122v1
• Date: Mon, 15 Jun 2026 02:28:34 GMT
• Title: Thinking with Visual Grounding
• Abstract summary: We introduce visually grounded thinking, a reasoning process in which models interleave natural-language thoughts with explicit point or box groundings of the visual evidence used at each step.<n>To train this behavior, we construct a scalable synthesis pipeline that distills correct visual reasoning traces.<n>We propose grounding-aware reinforcement learning, which combines answer correctness rewards with dense grounding rewards that score whether generated object references match the correct image evidence.
• Score: 55.778136782175615
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Visual thinking should not only sound right; it should show its evidence. While recent vision-language models (VLMs) can produce natural-language reasoning traces, these traces often leave the supporting image regions implicit, making them hard to verify and difficult to supervise. We introduce visually grounded thinking, a reasoning process in which models interleave natural-language thoughts with explicit point or box groundings of the visual evidence used at each step. This lets the model express intermediate reasoning in language while grounding key objects in the image regions they refer to. To train this behavior, we construct a scalable synthesis pipeline that distills correct visual reasoning traces, extracts the visual objects required by the traces, grounds them with a SAM3-based agent, and derives aligned point and box supervision from the resulting masks. We further propose grounding-aware reinforcement learning, which combines answer correctness rewards with dense grounding rewards that score whether generated object references match the correct image evidence. Across two counting benchmarks and four spatial reasoning benchmarks, adding visually grounded thinking to Gemma3-4B-IT consistently improves performance over the original model and the non-grounded thinking baseline. On spatial reasoning, the visually grounded thinking 4B models match, and in some cases surpass, Gemma3-27B-IT from the same model family. Our analysis shows that point grounding is well suited to counting, while box grounding benefits most from explicit grounding rewards on spatial tasks. Overall, our results show that VLMs think better when their intermediate thoughts are tied to the image regions that make them true.

Related papers

• How and What to Imagine? Visual Thinking in Unified Multimodal Models for Cross-View Spatial Reasoning [18.15809161363394]
  Cross-view spatial reasoning remains a weak spot for vision-language models.<n>We ask how to make visual thinking matter, and what kind of visual thinking works best.
  arXiv  Detail & Related papers  (2026-05-26T17:20:05Z)
• RieMind: Geometry-Grounded Spatial Agent for Scene Understanding [47.34079422330063]
  Current approaches rely on end-to-end video understanding or large-scale spatial question answering fine-tuning.<n>We propose an agentic framework for static 3D indoor scene reasoning that grounds an LLM in an explicit 3D scene graph (3DSG)<n>We find that our agentic variant achieves significantly better performance, with average improvements between 33% to 50%.
  arXiv  Detail & Related papers  (2026-03-16T15:02:24Z)
• Visual Reasoning Tracer: Object-Level Grounded Reasoning Benchmark [112.46338388724116]
  We introduce the Visual Reasoning Tracer (VRT) task, which requires models to not only localize the target object but also explicitly predict the intermediate objects that form the reasoning path.<n>To advance research in this area, we contribute: (1) VRT-Bench, a human-annotated benchmark for evaluating visual reasoning; (2) a new metric for assessing the quality of reasoning traces; and (3) VRT-80k, a large-scale dataset for reasoning model training.
  arXiv  Detail & Related papers  (2025-12-04T18:55:34Z)
• BLINK-Twice: You see, but do you observe? A Reasoning Benchmark on Visual Perception [67.89135437537179]
  We introduce BLINK-Twice, a vision-centric reasoning benchmark grounded in challenging perceptual tasks.<n>Instead of relying on external knowledge, our tasks require models to reason from visual content alone.<n>Compared to prior perception benchmarks, it moves beyond shallow perception and requires fine-grained observation and analytical reasoning.
  arXiv  Detail & Related papers  (2025-10-10T13:14:13Z)
• Traceable Evidence Enhanced Visual Grounded Reasoning: Evaluation and Methodology [87.65242416688146]
  TreeBench is a diagnostic benchmark for visual grounded reasoning.<n>TreeVGR is a training paradigm to supervise localization and reasoning jointly with reinforcement learning.
  arXiv  Detail & Related papers  (2025-07-10T17:59:58Z)
• Grounded Reinforcement Learning for Visual Reasoning [51.94871616778874]
  We introduce ViGoRL (Visually Grounded Reinforcement Learning), a vision-language model trained with reinforcement learning.<n>Inspired by human visual decision-making, ViGoRL learns to produce spatially grounded reasoning traces.<n>Our results show that visually grounded RL is a strong paradigm for imbuing models with general-purpose visual reasoning.
  arXiv  Detail & Related papers  (2025-05-29T17:20:26Z)
• v1: Learning to Point Visual Tokens for Multimodal Grounded Reasoning [27.688428439248607]
  We introduce v1, a lightweight extension that enables active visual referencing through a simple point-and-copy approach.<n>This allows the model to identify relevant image patches and copy their embeddings back into the reasoning stream.<n>Our pointing strategy lets the MLLM directly select image patches using their semantic representations as keys, keeping perceptual evidence embedded in the same space as the model's reasoning.
  arXiv  Detail & Related papers  (2025-05-24T19:30:47Z)
• Object-Centric Diagnosis of Visual Reasoning [118.36750454795428]
  This paper presents a systematical object-centric diagnosis of visual reasoning on grounding and robustness. We develop a diagnostic model, namely Graph Reasoning Machine. Our model replaces purely symbolic visual representation with probabilistic scene graph and then applies teacher-forcing training for the visual reasoning module.
  arXiv  Detail & Related papers  (2020-12-21T18:59:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.