AgentVLN: Towards Agentic Vision-and-Language Navigation

• URL: http://arxiv.org/abs/2603.17670v1
• Date: Wed, 18 Mar 2026 12:43:47 GMT
• Title: AgentVLN: Towards Agentic Vision-and-Language Navigation
• Abstract summary: Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments.<n>In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms.
• Score: 78.739525400071
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments. While Vision-Language Models (VLMs) offer strong 2D semantic understanding, current VLN systems remain constrained by limited spatial perception, 2D-3D representation mismatch, and monocular scale ambiguity. In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms. We formulate VLN as a Partially Observable Semi-Markov Decision Process (POSMDP) and introduce a VLM-as-Brain paradigm that decouples high-level semantic reasoning from perception and planning via a plug-and-play skill library. To resolve multi-level representation inconsistency, we design a cross-space representation mapping that projects perception-layer 3D topological waypoints into the image plane, yielding pixel-aligned visual prompts for the VLM. Building on this bridge, we integrate a context-aware self-correction and active exploration strategy to recover from occlusions and suppress error accumulation over long trajectories. To further address the spatial ambiguity of instructions in unstructured environments, we propose a Query-Driven Perceptual Chain-of-Thought (QD-PCoT) scheme, enabling the agent with the metacognitive ability to actively seek geometric depth information. Finally, we construct AgentVLN-Instruct, a large-scale instruction-tuning dataset with dynamic stage routing conditioned on target visibility. Extensive experiments show that AgentVLN consistently outperforms prior state-of-the-art methods (SOTA) on long-horizon VLN benchmarks, offering a practical paradigm for lightweight deployment of next-generation embodied navigation models. Code: https://github.com/Allenxinn/AgentVLN.

Related papers

• Traj-VLN: Learning Pixel-Space Interaction via Autoregressive Trajectory Generation [5.0211517769291065]
  We propose fine-tuning vision-language models to learn navigation interactions directly in 2D pixel space.<n>Our model sequentially predicts a series of pixel coordinates, drawing a trajectory from the bottom center of the current observation.
  arXiv  Detail & Related papers  (2026-07-12T12:54:44Z)
• Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation [32.02017382315305]
  Vision-Language Navigation (VLN) enables embodied agents to reach target locations in unseen environments by following language instructions.<n>Despite recent progress with vision-language models (VLMs), a critical semantic-geometric gap remains.<n>We propose a Hierarchical Semantic-Geometric Map (HSGM) that transforms 3D geometric information into a structured representation compatible withVLMs.
  arXiv  Detail & Related papers  (2026-05-25T08:53:21Z)
• AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation [78.80536515102305]
  Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment.<n>We propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism.<n>Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training.
  arXiv  Detail & Related papers  (2026-05-21T17:58:26Z)
• SEDualVLN: A Spatially-Enhanced Dual-System for Vision-Language Navigation [19.9336847486232]
  We introduce SEDualVLN, a spatially-enhanced dual-system Vision-Language Navigation framework.<n>System 1 is a VLM model enhanced with both global and local spatial awareness, used for action generation.<n>System 2 integrates a general MLLM with a mapping module, wherein the MLLM plans waypoints by leveraging top-down views of the real-time 3D map.
  arXiv  Detail & Related papers  (2026-05-17T04:12:56Z)
• DyGeoVLN: Infusing Dynamic Geometry Foundation Model into Vision-Language Navigation [19.94547112119204]
  Vision-language Navigation (VLN) requires an agent to understand visual observations and language instructions to navigate in unseen environments.<n>Most existing approaches rely on static scene assumptions and struggle to generalize in dynamic, real-world scenarios.<n>We propose DyGeoVLN, a dynamic geometry-aware VLN framework.
  arXiv  Detail & Related papers  (2026-03-22T14:56:59Z)
• OmniVLN: Omnidirectional 3D Perception and Token-Efficient LLM Reasoning for Visual-Language Navigation across Air and Ground Platforms [33.40889181799252]
  Language-guided embodied navigation requires an agent to interpret object-referential instructions, search across multiple rooms, localize the referenced target, and execute reliable motion toward it.<n>We present OmniVLN, a zero-shot visual-language navigation framework that couples omnidirectional 3D perception with token-efficient hierarchical reasoning for both aerial and ground robots.<n> Experiments show that the proposed hierarchical interface improves spatial referring accuracy from 77.27% to 93.18%, reduces cumulative prompt tokens by up to 61.7% in cluttered multi-room settings, and improves navigation success by up to 11.68% over a flat-
  arXiv  Detail & Related papers  (2026-03-18T04:26:30Z)
• TagaVLM: Topology-Aware Global Action Reasoning for Vision-Language Navigation [70.23578202012048]
  Vision-Language Navigation (VLN) presents a unique challenge for Large Vision-Language Models (VLMs) due to their inherent architectural mismatch.<n>We propose TagaVLM (Topology-Aware Global Action reasoning), an end-to-end framework that explicitly injects topological structures into the VLM backbone.<n>To enhance topological node information, an Interleaved Navigation Prompt strengthens node-level visual-text alignment.<n>With the embedded topological graph, the model is capable of global action reasoning, allowing for robust path correction.
  arXiv  Detail & Related papers  (2026-03-03T13:28:07Z)
• 3DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splatting [12.057873540714098]
  3DGSNav is a novel framework that embeds 3D Gaussian Splatting (3DGS) as persistent memory for vision-language models (VLMs) to enhance spatial reasoning.<n>3DGSNav incrementally constructs a 3DGS representation of the environment, enabling trajectory-guided free-viewpoint rendering of frontier-aware first-person views.<n>During navigation, a real-time object detector filters potential targets, while VLM-driven active viewpoint switching performs target re-verification.
  arXiv  Detail & Related papers  (2026-02-12T16:41:26Z)
• A Navigation Framework Utilizing Vision-Language Models [0.0]
  Vision-and-Language Navigation (VLN) presents a complex challenge in embodied AI.<n>Recent advances in large vision-language models (LVLMs), such as CLIP and Flamingo, have significantly improved multimodal understanding.<n>We propose a modular, plug-and-play navigation framework that decouples vision-language understanding from action planning.
  arXiv  Detail & Related papers  (2025-06-11T20:51:58Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)
• UnitedVLN: Generalizable Gaussian Splatting for Continuous Vision-Language Navigation [71.97405667493477]
  We introduce a novel, generalizable 3DGS-based pre-training paradigm, called UnitedVLN.<n>It enables agents to better explore future environments by unitedly rendering high-fidelity 360 visual images and semantic features.<n>UnitedVLN outperforms state-of-the-art methods on existing VLN-CE benchmarks.
  arXiv  Detail & Related papers  (2024-11-25T02:44:59Z)
• BEVBert: Multimodal Map Pre-training for Language-guided Navigation [75.23388288113817]
  We propose a new map-based pre-training paradigm that is spatial-aware for use in vision-and-language navigation (VLN) We build a local metric map to explicitly aggregate incomplete observations and remove duplicates, while modeling navigation dependency in a global topological map. Based on the hybrid map, we devise a pre-training framework to learn a multimodal map representation, which enhances spatial-aware cross-modal reasoning thereby facilitating the language-guided navigation goal.
  arXiv  Detail & Related papers  (2022-12-08T16:27:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.