See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay

• URL: http://arxiv.org/abs/2603.11601v1
• Date: Thu, 12 Mar 2026 06:48:57 GMT
• Title: See, Symbolize, Act: Grounding VLMs with Spatial Representations for Better Gameplay
• Abstract summary: Vision-Language Models (VLMs) excel at describing visual scenes, yet struggle to translate perception into precise, grounded actions.<n>We investigate whether providing VLMs with both the visual frame and the symbolic representation of the scene can improve their performance in interactive environments.<n>We evaluate three state-of-the-art VLMs across Atari games, VizDoom, and AI2-THOR, comparing frame-only, frame with self-extracted symbols, frame with ground-truth symbols, and symbol-only pipelines.
• Score: 1.0742675209112622
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Models (VLMs) excel at describing visual scenes, yet struggle to translate perception into precise, grounded actions. We investigate whether providing VLMs with both the visual frame and the symbolic representation of the scene can improve their performance in interactive environments. We evaluate three state-of-the-art VLMs across Atari games, VizDoom, and AI2-THOR, comparing frame-only, frame with self-extracted symbols, frame with ground-truth symbols, and symbol-only pipelines. Our results indicate that all models benefit when the symbolic information is accurate. However, when VLMs extract symbols themselves, performance becomes dependent on model capability and scene complexity. We further investigate how accurately VLMs can extract symbolic information from visual inputs and how noise in these symbols affects decision-making and gameplay performance. Our findings reveal that symbolic grounding is beneficial in VLMs only when symbol extraction is reliable, and highlight perception quality as a central bottleneck for future VLM-based agents.

Related papers

• The Hidden Evolution of Disguised Visual Context inside the VLM [8.702550456024769]
  We show that visual tokens enter Large Language Models (LLMs) as raw, foreign signals.<n>We evaluate in-context and layer-wise injection VLM integration paradigms under identical training conditions.<n>We show that this evolution determines what visual features the VLM can utilize effectively.
  arXiv  Detail & Related papers  (2026-06-18T10:52:49Z)
• From Frames to Temporal Graphs: In-Context Egocentric Action Recognition with Vision-Language Models [67.00247936789127]
  Action reasoning in egocentric video requires capturing fine-grained transitions of hand-object interactions.<n>We propose to decouple visual perception from symbolic reasoning by converting videos into Temporal Action Graphs.
  arXiv  Detail & Related papers  (2026-06-13T18:00:42Z)
• Aligning What Vision-Language Models See and Perceive with Adaptive Information Flow [32.21470980856891]
  Vision-Language Models (VLMs) have demonstrated strong capability in a wide range of tasks such as visual recognition, document parsing, and visual grounding.<n>Recent work shows that while VLMs often manage to capture the correct image region corresponding to the question, they do not necessarily produce the correct answers.<n>We show that this misalignment could be attributed to suboptimal information flow within VLMs, where text tokens distribute too much attention to irrelevant visual tokens.
  arXiv  Detail & Related papers  (2026-04-17T08:07:22Z)
• Hidden in plain sight: VLMs overlook their visual representations [48.83628674170634]
  We compare vision language models (VLMs) to their visual encoders to understand their ability to integrate across these modalities.<n>We find that VLMs perform substantially worse than their visual encoders, dropping to near-chance performance.
  arXiv  Detail & Related papers  (2025-06-09T17:59:54Z)
• Vision language models are unreliable at trivial spatial cognition [0.2902243522110345]
  Vision language models (VLMs) are designed to extract relevant visuospatial information from images.<n>We develop a benchmark dataset -- TableTest -- whose images depict 3D scenes of objects arranged on a table, and used it to evaluate state-of-the-art VLMs.<n>Results show that performance could be degraded by minor variations of prompts that use equivalent descriptions.
  arXiv  Detail & Related papers  (2025-04-22T17:38:01Z)
• Introducing Visual Perception Token into Multimodal Large Language Model [53.82301522384719]
  Multimodal Large Language Model (MLLM) relies on the perception process of its vision encoder.<n>MLLM still lacks the autonomous capability to control its own visual perception processes.<n>We propose the concept of Visual Perception Token, aiming to empower MLLM with a mechanism to control its visual perception processes.
  arXiv  Detail & Related papers  (2025-02-24T18:56:12Z)
• Can Large Language Models Understand Symbolic Graphics Programs? [136.5639211254501]
  Symbolic graphics programs are popular in computer graphics.<n>They allow us to test an LLM's ability to answer semantic questions about the images or 3D geometries without a vision encoder.<n>We create a benchmark for the semantic visual understanding of symbolic graphics programs, built procedurally with minimal human effort.<n>We evaluate commercial and open-source LLMs on our benchmark to assess their ability to reason about visual output of programs.
  arXiv  Detail & Related papers  (2024-08-15T17:59:57Z)
• How Well Can Vision Language Models See Image Details? [53.036922527685064]
  We introduce a pixel value prediction task to explore "How Well Can Vision Language Models See Image Details?" Our research reveals that incorporating pixel value prediction as one of the VLM pre-training tasks and vision encoder adaptation markedly boosts VLM performance on downstream image-language understanding tasks.
  arXiv  Detail & Related papers  (2024-08-07T17:59:40Z)
• BEAF: Observing BEfore-AFter Changes to Evaluate Hallucination in Vision-language Models [20.697019266074747]
  Vision language models (VLMs) perceive the world through a combination of a visual encoder and a large language model (LLM) Recent studies show that VLMs are vulnerable to hallucination. We introduce new metrics: True Understanding (TU), IGnorance (IG), StuBbornness (SB), and InDecision (ID)
  arXiv  Detail & Related papers  (2024-07-18T12:11:12Z)
• Visually Descriptive Language Model for Vector Graphics Reasoning [76.42082386029206]
  We propose the Visually Descriptive Language Model (VDLM) to bridge the gap between low-level visual perception and high-level language reasoning.<n>We show that VDLM significantly improves state-of-the-art LMMs like GPT-4o on various multimodal perception and reasoning tasks.
  arXiv  Detail & Related papers  (2024-04-09T17:30:18Z)
• If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions [9.190831897944957]
  Vision-Language Model (VLM) representations are often based on visual attributes like shape.<n>We propose Extract and Explore (EX2), a novel approach to characterize textual features that are important for VLM.<n>We show that EX2 uses reinforcement learning to align a large language model with VLM preferences and generates descriptions that incorporate features that are important for the VLM.
  arXiv  Detail & Related papers  (2024-03-25T06:05:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.