What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity

• URL: http://arxiv.org/abs/2605.03782v1
• Date: Tue, 05 May 2026 14:08:54 GMT
• Title: What You Think is What You See: Driving Exploration in VLM Agents via Visual-Linguistic Curiosity
• Abstract summary: We ask: Can VLM agents actively find signals that challenge and refine their internal world model through curiosity-driven exploration?<n>We propose GLANCE, a unified framework that bridges reasoning and exploration by grounding the agent's linguistic world model into the stable visual representations of an evolving target network.
• Score: 38.38728887407681
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: To navigate partially observable visual environments, recent VLM agents increasingly internalize world modeling capabilities into their policies via explicit CoT reasoning, enabling them to mentally simulate futures before acting. However, relying solely on passive reasoning over visited states is insufficient for sparse-reward tasks, as it lacks the epistemic drive to actively uncover the ``known unknown'' required for robust generalization. We ask: Can VLM agents actively find signals that challenge and refine their internal world model through curiosity-driven exploration? In this work, we propose GLANCE, a unified framework that bridges reasoning and exploration by grounding the agent's linguistic world model into the stable visual representations of an evolving target network. Crucially, GLANCE leverages the discrepancy between linguistic prediction and visual reality as an intrinsic curiosity signal within reinforcement learning, steering the agent to actively explore areas where its internal model is uncertain. Extensive experiments across a series of agentic tasks show the effectiveness of GLANCE, and demonstrate that aligning ``what the agent thinks'' with ``what the agent sees'' is key to solving complex or sparse agentic tasks.

Related papers

• Towards Generalizable Visually Grounded Exploration of Household Devices [74.66117923766632]
  Key bottleneck of realizing general autonomous embodied agents lies in Generalizable Visually Grounded Exploration.<n>We introduce VGEBench, a benchmark designed to evaluate the generalizable visually grounded exploration capabilities of Vision-Language Models.<n>Unlike static datasets, we construct a Logic-Driven State Machine framework. This framework simulates multi-turn interaction loops, compelling agents to achieve goals by active visual perception and feedback-driven correction.
  arXiv  Detail & Related papers  (2026-09-01T07:42:45Z)
• Multi-Agent LLMs Fail to Explore Each Other [34.74439361133657]
  We show that modern large language model (LLM) agents fail to explore effectively when interacting with one another.<n>We introduce Multi- Agent Contextual Exploration (MACE), a lightweight framework that explicitly promotes exploration through structured peer selection.
  arXiv  Detail & Related papers  (2026-07-13T08:34:05Z)
• Playing ZendoWorld: Challenging AI Agents on Active Visual Concept Induction [39.71864879537297]
  ZendoWorld is a controlled interactive environment in which agents must infer a logical rule about visual game observations.<n>We evaluate several agents spanning pure VLM reasoning, Bayesian particle filtering, dynamic concept discovery, and neuro-symbolic methods.<n>Our main findings are: (1) high accuracy in predicting labels does not imply recovery of the underlying rule; (2) perception and induction are distinct bottlenecks for different agent classes; and (3) VLM-based agents propose near-uninformative experiments, failing to actively reduce hypothesis uncertainty.
  arXiv  Detail & Related papers  (2026-07-09T08:29:23Z)
• Remember to be Curious: Episodic Context and Persistent Worlds for 3D Exploration [60.63663437013685]
  In this work, we demonstrate that this failure stems from a lack of spatial persistence and episodic context.<n>We achieve this using an online 3D reconstruction as a persistent model of the world, while the agent policy is parameterized as a sequence model over RGB observations to maintain episodic context.<n>Our end-to-end policy enables efficient adaptation to downstream tasks, such as apple picking and image-goal navigation, outperforming from-scratch baselines.
  arXiv  Detail & Related papers  (2026-05-21T17:58:06Z)
• Theory of Space: Can Foundation Models Construct Spatial Beliefs through Active Exploration? [83.13508919229939]
  Theory of Space is defined as an agent's ability to actively acquire information through self-directed, active exploration.<n>A key innovation is spatial belief probing, which prompts models to reveal their internal spatial representations at each step.<n>Our findings suggest that current foundation models struggle to maintain coherent, revisable spatial beliefs during active exploration.
  arXiv  Detail & Related papers  (2026-02-04T19:06:40Z)
• Imagine, Verify, Execute: Memory-guided Agentic Exploration with Vision-Language Models [81.08295968057453]
  We present IVE, an agentic exploration framework inspired by human curiosity.<n>We evaluate IVE in both simulated and real-world tabletop environments.
  arXiv  Detail & Related papers  (2025-05-12T17:59:11Z)
• SENSEI: Semantic Exploration Guided by Foundation Models to Learn Versatile World Models [22.96777963013918]
  Intrinsic motivation attempts to decouple exploration from external, task-based rewards.<n>SENSEI is a framework to equip model-based RL agents with an intrinsic motivation for semantically meaningful behavior.
  arXiv  Detail & Related papers  (2025-03-03T14:26:15Z)
• DREAMWALKER: Mental Planning for Continuous Vision-Language Navigation [107.5934592892763]
  We propose DREAMWALKER -- a world model based VLN-CE agent. The world model is built to summarize the visual, topological, and dynamic properties of the complicated continuous environment. It can simulate and evaluate possible plans entirely in such internal abstract world, before executing costly actions.
  arXiv  Detail & Related papers  (2023-08-14T23:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.