Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text

• URL: http://arxiv.org/abs/2607.21072v1
• Date: Thu, 23 Jul 2026 09:04:48 GMT
• Title: Show, Don't Tell: Evaluating Spatial Cognition in Generative Pixels Rather Than LLM Text
• Abstract summary: spatial intelligence is essential for agents to move from static semantic understanding toward interacting with the physical world.<n>Existing spatial reasoning benchmarks usually require coordinates, options, or text, creating an answer-interface mismatch for image-generation models.<n>We propose ProVisE, a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models.
• Score: 19.512871070260477
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Spatial intelligence is essential for agents to move from static semantic understanding toward interacting with the physical world. Many spatial tasks are grounded in continuous visual scenes, where locations, regions, and paths are more naturally expressed by pointing, marking, or drawing than by reporting precise coordinates or discrete textual symbols. Yet existing spatial reasoning benchmarks usually require coordinates, options, or text, creating an answer-interface mismatch for image-generation models. This makes it difficult to evaluate image-generation models under the same task semantics as text-output VLMs, despite their ability to externalize spatial judgments directly in pixel space. We propose ProVisE (Protocolized Visual Evaluation), a benchmark-agnostic framework that elicits protocol-constrained visual answers from image-generation models and parses them into structured predictions compatible with original metrics. ProVisE also includes an Agentic builder that constructs and validates task-specific protocols for new benchmarks. We further introduce SpatialGen-Bench, a curated diagnostic benchmark of 470 samples across 14 spatial subtasks, four capability levels, and diverse answer forms. We evaluate representative text-output VLMs and image-generation models in a unified setting and validate Agentic protocol construction on six external spatial benchmarks. Results show that image-generation models are competitive when spatial answers can be externalized directly in pixel space, while text-output VLMs retain a clear advantage in compositional spatial reasoning. These findings reveal complementary strengths of pixel-space expression and text-based reasoning and establish a metric-compatible testbed for studying spatial cognition in image-generation models.

Related papers

• Vision as Unified Multimodal Generation [73.31482353559736]
  SenseNova-Vision is a computer-vision instruction-response corpus spanning text, image, and mixed targets.<n>The model covers a broad range of vision tasks, including detection, OCR, keypoint estimation, segmentation, depth estimation, surface normal prediction, point maps, and camera pose estimation.<n> Experiments show that a single unified model can match leading task-specialized systems across structured visual understanding, dense geometric prediction, segmentation, and multi-view visual geometry.
  arXiv  Detail & Related papers  (2026-07-07T17:58:33Z)
• Probing Geospatial SSL Representations with Environmental Signals [51.908606610432194]
  Self-supervised learning (SSL) is designed to learn generic, transferable representations rather than representations optimized for a single task.<n>We ask a different question: do SSL representations preserve statistical associations with environmental variables that co-vary with the imaging process?<n>To answer this question, we probe SSL representations using co-located ERA5 reanalysis variables, a global dataset of physically consistent environmental variables.
  arXiv  Detail & Related papers  (2026-07-06T15:24:25Z)
• OmniGF: A Dual-Branch Vision-Language Framework for Unified Gaze Following [59.53720386342017]
  We propose a unified vision-language framework that adapts foundational Vision-Language Models for scalable multi-person gaze reasoning.<n>By modeling all individuals, OmniGF seamlessly integrates precise spatial gaze target estimation, semantic gaze prediction, and complex social gaze reasoning.
  arXiv  Detail & Related papers  (2026-05-26T00:08:06Z)
• SpatialReward: Verifiable Spatial Reward Modeling for Fine-Grained Spatial Consistency in Text-to-Image Generation [62.55421542903781]
  We present textbfSpatialReward, a verifiable reward model explicitly designed to evaluate spatial layouts in generated images.<n>Experiments on Stable Diffusion and FLUX show that incorporating SpatialReward into RL training consistently improves spatial consistency and overall generation quality.
  arXiv  Detail & Related papers  (2026-03-23T17:26:35Z)
• Uni-RS: A Spatially Faithful Unified Understanding and Generation Model for Remote Sensing [9.357861053928898]
  Unified remote sensing multimodal models exhibit a pronounced spatial reversal curse.<n>We propose Uni-RS, the first unified model tailored for remote sensing.<n>We show that our approach substantially improves spatial faithfulness in text-to-image generation.
  arXiv  Detail & Related papers  (2026-01-25T03:22:26Z)
• CoPatch: Zero-Shot Referring Image Segmentation by Leveraging Untapped Spatial Knowledge in CLIP [26.827036116024914]
  textscCoPatch is a zero-shot RIS framework that enhances spatial representations in both text and image modalities.<n>We show that textscCoPatch significantly improves spatial grounding in zero-shot RIS across RefCOCO, RefCOCO+, RefCOCOg, and PhraseCut (+ 2--7 mIoU) without requiring any additional training.
  arXiv  Detail & Related papers  (2025-09-27T04:12:10Z)
• Mind-the-Glitch: Visual Correspondence for Detecting Inconsistencies in Subject-Driven Generation [120.23172120151821]
  We propose a novel approach for disentangling visual and semantic features from the backbones of pre-trained diffusion models.<n>We introduce an automated pipeline that constructs image pairs with annotated semantic and visual correspondences.<n>We propose a new metric, Visual Semantic Matching, that quantifies visual inconsistencies in subject-driven image generation.
  arXiv  Detail & Related papers  (2025-09-26T07:11:55Z)
• Why Settle for Mid: A Probabilistic Viewpoint to Spatial Relationship Alignment in Text-to-image Models [3.5999252362400993]
  A prevalent issue in compositional generation is the misalignment of spatial relationships.<n>We introduce a novel evaluation metric designed to assess the alignment of 2D and 3D spatial relationships between text and image.<n>We also propose PoS-based Generation, an inference-time method that improves the alignment of 2D and 3D spatial relationships in T2I models without requiring fine-tuning.
  arXiv  Detail & Related papers  (2025-06-29T22:41:27Z)
• MPDrive: Improving Spatial Understanding with Marker-Based Prompt Learning for Autonomous Driving [23.012850820921592]
  Marker-based Prompt learning framework (MPDrive) represents spatial coordinates by concise visual markers.<n>MPDrive achieves state-of-the-art performance, particularly in cases requiring sophisticated spatial understanding.
  arXiv  Detail & Related papers  (2025-04-01T02:49:39Z)
• Mind the Gap: Benchmarking Spatial Reasoning in Vision-Language Models [14.442394137843923]
  We present a detailed analysis that first delineates the core elements of spatial reasoning.<n>We then assesses the performance of these models in both synthetic and real-world images.
  arXiv  Detail & Related papers  (2025-03-25T14:34:06Z)
• REVISION: Rendering Tools Enable Spatial Fidelity in Vision-Language Models [67.55362046790512]
  Vision-language models lack the ability to correctly reason over spatial relationships. We develop the REVISION framework which improves spatial fidelity in vision-language models. Our results and findings indicate that utilizing rendering-based frameworks is an effective approach for developing spatially-aware models.
  arXiv  Detail & Related papers  (2024-08-05T04:51:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.