Recursive Visual Imagination and Adaptive Linguistic Grounding for Vision Language Navigation

• URL: http://arxiv.org/abs/2507.21450v1
• Date: Tue, 29 Jul 2025 02:40:07 GMT
• Title: Recursive Visual Imagination and Adaptive Linguistic Grounding for Vision Language Navigation
• Authors: Bolei Chen, Jiaxu Kang, Yifei Wang, Ping Zhong, Qi Wu, Jianxin Wang,
• Abstract summary: Vision Language Navigation (VLN) typically requires agents to navigate to specified objects or remote regions in unknown scenes by obeying linguistic commands.<n>Current agents suffer from overly detailed scene representation and ambiguous vision-language alignment.<n>We propose a navigation policy by summarizing along-the-way visual perceptions, which are adaptively aligned with commands to enhance linguistic grounding.
• Score: 25.111153186227728
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision Language Navigation (VLN) typically requires agents to navigate to specified objects or remote regions in unknown scenes by obeying linguistic commands. Such tasks require organizing historical visual observations for linguistic grounding, which is critical for long-sequence navigational decisions. However, current agents suffer from overly detailed scene representation and ambiguous vision-language alignment, which weaken their comprehension of navigation-friendly high-level scene priors and easily lead to behaviors that violate linguistic commands. To tackle these issues, we propose a navigation policy by recursively summarizing along-the-way visual perceptions, which are adaptively aligned with commands to enhance linguistic grounding. In particular, by structurally modeling historical trajectories as compact neural grids, several Recursive Visual Imagination (RVI) techniques are proposed to motivate agents to focus on the regularity of visual transitions and semantic scene layouts, instead of dealing with misleading geometric details. Then, an Adaptive Linguistic Grounding (ALG) technique is proposed to align the learned situational memories with different linguistic components purposefully. Such fine-grained semantic matching facilitates the accurate anticipation of navigation actions and progress. Our navigation policy outperforms the state-of-the-art methods on the challenging VLN-CE and ObjectNav tasks, showing the superiority of our RVI and ALG techniques for VLN.

Related papers

• UAV-VLN: End-to-End Vision Language guided Navigation for UAVs [0.0]
  A core challenge in AI-guided autonomy is enabling agents to navigate realistically and effectively in previously unseen environments.<n>We propose UAV-VLN, a novel end-to-end Vision-Language Navigation framework for Unmanned Aerial Vehicles (UAVs)<n>Our system interprets free-form natural language instructions, grounds them into visual observations, and plans feasible aerial trajectories in diverse environments.
  arXiv  Detail & Related papers  (2025-04-30T08:40:47Z)
• LangNav: Language as a Perceptual Representation for Navigation [63.90602960822604]
  We explore the use of language as a perceptual representation for vision-and-language navigation (VLN) Our approach uses off-the-shelf vision systems for image captioning and object detection to convert an agent's egocentric panoramic view at each time step into natural language descriptions.
  arXiv  Detail & Related papers  (2023-10-11T20:52:30Z)
• Learning Navigational Visual Representations with Semantic Map Supervision [85.91625020847358]
  We propose a navigational-specific visual representation learning method by contrasting the agent's egocentric views and semantic maps. Ego$2$-Map learning transfers the compact and rich information from a map, such as objects, structure and transition, to the agent's egocentric representations for navigation.
  arXiv  Detail & Related papers  (2023-07-23T14:01:05Z)
• VELMA: Verbalization Embodiment of LLM Agents for Vision and Language Navigation in Street View [81.58612867186633]
  Vision and Language Navigation(VLN) requires visual and natural language understanding as well as spatial and temporal reasoning capabilities. We show that VELMA is able to successfully follow navigation instructions in Street View with only two in-context examples. We further finetune the LLM agent on a few thousand examples and achieve 25%-30% relative improvement in task completion over the previous state-of-the-art for two datasets.
  arXiv  Detail & Related papers  (2023-07-12T11:08:24Z)
• ESceme: Vision-and-Language Navigation with Episodic Scene Memory [72.69189330588539]
  Vision-and-language navigation (VLN) simulates a visual agent that follows natural-language navigation instructions in real-world scenes. We introduce a mechanism of Episodic Scene memory (ESceme) for VLN that wakes an agent's memories of past visits when it enters the current scene.
  arXiv  Detail & Related papers  (2023-03-02T07:42:07Z)
• CLEAR: Improving Vision-Language Navigation with Cross-Lingual, Environment-Agnostic Representations [98.30038910061894]
  Vision-and-Language Navigation (VLN) tasks require an agent to navigate through the environment based on language instructions. We propose CLEAR: Cross-Lingual and Environment-Agnostic Representations. Our language and visual representations can be successfully transferred to the Room-to-Room and Cooperative Vision-and-Dialogue Navigation task.
  arXiv  Detail & Related papers  (2022-07-05T17:38:59Z)
• Know What and Know Where: An Object-and-Room Informed Sequential BERT for Indoor Vision-Language Navigation [120.90387630691816]
  Vision-and-Language Navigation (VLN) requires an agent to navigate to a remote location on the basis of natural-language instructions and a set of photo-realistic panoramas. Most existing methods take words in instructions and discrete views of each panorama as the minimal unit of encoding. We propose an object-informed sequential BERT to encode visual perceptions and linguistic instructions at the same fine-grained level.
  arXiv  Detail & Related papers  (2021-04-09T02:44:39Z)
• Structured Scene Memory for Vision-Language Navigation [155.63025602722712]
  We propose a crucial architecture for vision-language navigation (VLN) It is compartmentalized enough to accurately memorize the percepts during navigation. It also serves as a structured scene representation, which captures and disentangles visual and geometric cues in the environment.
  arXiv  Detail & Related papers  (2021-03-05T03:41:00Z)
• A modular vision language navigation and manipulation framework for long horizon compositional tasks in indoor environment [9.159670926457975]
  MoViLan is a new framework for execution of visually grounded natural language instructions. We propose a modular approach to deal with the combined navigation and object interaction problem. Specifically, we propose a novel geometry-aware mapping technique for cluttered indoor environments.
  arXiv  Detail & Related papers  (2021-01-19T23:05:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.