SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimization

• URL: http://arxiv.org/abs/2512.02631v1
• Date: Tue, 02 Dec 2025 10:40:46 GMT
• Title: SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimization
• Abstract summary: Existing Vision-Language Navigation (VLN) agents often suffer from perception errors, reasoning errors, and planning errors.<n>To address these limitations, a novel VLN agent framework, named SeeNav-Agent, is proposed in this work.<n>Step Reward Group Policy Optimization (SRGPO) is designed for the post-training of VLN agents.
• Score: 32.785475974900244
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing Vision-Language Navigation (VLN) agents based on Large Vision-Language Models (LVLMs) often suffer from perception errors, reasoning errors, and planning errors, which significantly hinder their navigation performance. To address these limitations, a novel VLN agent framework, named SeeNav-Agent, is proposed in this work. First, to reduce perception hallucinations of the visual module of the VLN agent, a dual-view Visual Prompt (VP) technique is introduced in the input space, which can also improve the agent's understanding of current spatial states. Subsequently, a novel step-level Reinforcement Fine-Tuning (RFT) method, Step Reward Group Policy Optimization (SRGPO), is designed for the post-training of VLN agents. In SRGPO, we first define verifiable process rewards for the navigation task, and then perform efficient step-level advantage estimation by randomly grouping different navigation steps. SRGPO provides dense reward signals for the reinforcement learning process of the VLN agent and enhances its planning capability. Experimental results on the EmbodiedBench Navigation benchmark indicate that by introducing the zero-shot VP module, the GPT-4.1 achieves a navigation success rate of 86.7%, surpassing the current best LVLM by approximately 20 percentage points (pp). Through post-training based on SRGPO, the Qwen2.5-VL-3B model reaches a navigation success rate of 72.3%, outperforming the best existing LVLM model by 5.6 pp. Moreover, compared to RFT algorithms such as GRPO and GiGPO, the proposed SRGPO demonstrates significant improvements in training stability, convergence efficiency, and generalization capability.

Related papers

• VerNav: Verifier-First Low-Latency Vision-and-Language Navigation [7.190348268664948]
  Vision-and-Language Navigation (VLN) requires an agent to navigate through unseen 3D environments according to natural-latency instructions.<n>We propose VerNav, a verifier-first framework for low-language LLM-based VLN.
  arXiv  Detail & Related papers  (2026-09-01T08:45:15Z)
• PGN: Design and Implementation of a Vision-Language Navigation System Based on Pangu Multimodal Foundation Model [10.02560644098367]
  Vision-Language Navigation (VLN) requires an embodied agent to interpret a natural computation instruction and predict actions from temporally ordered visual observations.<n>This report presents PGN (Pangu Navigator), an offline VLN action-prediction system built on OpenPangu-7B.
  arXiv  Detail & Related papers  (2026-07-20T10:48:48Z)
• Joint On-and-Off Policy Learning for Vision-and-Language Navigation [44.53787822402111]
  Vision-and-Language Navigation (VLN) requires an embodied agent to navigate in the physical world by adhering to natural language instructions.<n>Recent advancements in Vision-Language Models (VLM) have propelled the development of VLM-based VLN methods.<n>This paper introduces JOP-VLN, a novel VLN framework that combines off-policy imitation learning and on-policy exploration within a three-stage training pipeline.
  arXiv  Detail & Related papers  (2026-07-15T05:46:34Z)
• Trajectory-Diversity-Driven Robust Vision-and-Language Navigation [39.4949197412916]
  Vision-and-Language Navigation (VLN) requires agents to navigate photo-realistic environments following natural language instructions.<n>We present NavGRPO, a reinforcement learning framework that learns goal-directed navigation policies.<n>We demonstrate +14.89% SPL gain over the baseline, confirming that goal-directed RL training builds substantially more robust navigation policies.
  arXiv  Detail & Related papers  (2026-03-16T14:46:54Z)
• Vision Language Models for Optimization-Driven Intent Processing in Autonomous Networks [6.377294874945291]
  Intent-Based Networking (IBN) allows operators to specify high-level network goals rather than low-level configurations.<n>Current systems assume text-based intent expression, requiring operators to enumerate topologies and parameters in prose.<n>We show Vision-Language Models (VLMs) can process annotated network sketches into correct optimization code.
  arXiv  Detail & Related papers  (2026-01-19T05:57:58Z)
• ActiveVLN: Towards Active Exploration via Multi-Turn RL in Vision-and-Language Navigation [57.399685080574756]
  Existing MLLM-based VLN methods rely on imitation learning (IL) and often use DAgger for post-training.<n>We propose ActiveVLN, a VLN framework that explicitly enables active exploration through multi-turn RL.<n>Experiments show that ActiveVLN achieves the largest performance gains over IL baselines compared to both DAgger-based and prior RL-based post-training methods.
  arXiv  Detail & Related papers  (2025-09-16T03:31:46Z)
• DAgger Diffusion Navigation: DAgger Boosted Diffusion Policy for Vision-Language Navigation [73.80968452950854]
  Vision-Language Navigation in Continuous Environments (VLN-CE) requires agents to follow natural language instructions through free-form 3D spaces.<n>Existing VLN-CE approaches typically use a two-stage waypoint planning framework.<n>We propose DAgger Diffusion Navigation (DifNav) as an end-to-end optimized VLN-CE policy.
  arXiv  Detail & Related papers  (2025-08-13T02:51:43Z)
• Harnessing Input-Adaptive Inference for Efficient VLN [13.847596428283861]
  An emerging paradigm in vision-and-language navigation (VLN) is the use of history-aware multi-modal transformer models.<n>We propose a novel input-adaptive navigation method to enhance VLN model efficiency.
  arXiv  Detail & Related papers  (2025-08-12T18:05:33Z)
• VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning [77.34267241692706]
  Vision-Language Navigation (VLN) is a core challenge in embodied AI, requiring agents to navigate real-world environments using natural language instructions.<n>We propose VLN-R1, an end-to-end framework that leverages Large Vision-Language Models (LVLM) to directly translate egocentric video streams into continuous navigation actions.
  arXiv  Detail & Related papers  (2025-06-20T17:59:59Z)
• EvolveNav: Empowering LLM-Based Vision-Language Navigation via Self-Improving Embodied Reasoning [145.32076310071434]
  We propose EvolveNav, a novel embodied reasoning paradigm that realizes adaptable and generalizable navigational reasoning.<n>EvolveNav involves a two-stage training process: (1) Formalized CoT Supervised Fine-Tuning, where we train the model with curated formalized CoT labels to first activate the model's navigational reasoning capabilities, and simultaneously increase the reasoning speed; (2) Self-Reflective Post-Training, where the model is iteratively trained with its own reasoning outputs as self-enriched CoT labels to enhance the supervision diversity.
  arXiv  Detail & Related papers  (2025-06-02T11:28:32Z)
• Affordances-Oriented Planning using Foundation Models for Continuous Vision-Language Navigation [64.84996994779443]
  We propose a novel Affordances-Oriented Planner for continuous vision-language navigation (VLN) task. Our AO-Planner integrates various foundation models to achieve affordances-oriented low-level motion planning and high-level decision-making. Experiments on the challenging R2R-CE and RxR-CE datasets show that AO-Planner achieves state-of-the-art zero-shot performance.
  arXiv  Detail & Related papers  (2024-07-08T12:52:46Z)
• NavCoT: Boosting LLM-Based Vision-and-Language Navigation via Learning Disentangled Reasoning [97.88246428240872]
  Vision-and-Language Navigation (VLN), as a crucial research problem of Embodied AI, requires an embodied agent to navigate through complex 3D environments following natural language instructions.<n>Recent research has highlighted the promising capacity of large language models (LLMs) in VLN by improving navigational reasoning accuracy and interpretability.<n>This paper introduces a novel strategy called Navigational Chain-of-Thought (NavCoT), where we fulfill parameter-efficient in-domain training to enable self-guided navigational decision.
  arXiv  Detail & Related papers  (2024-03-12T07:27:02Z)
• NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models [17.495162643127003]
  We introduce the NavGPT to reveal the reasoning capability of GPT models in complex embodied scenes. NavGPT takes the textual descriptions of visual observations, navigation history, and future explorable directions as inputs to reason the agent's current status. We show that NavGPT is capable of generating high-quality navigational instructions from observations and actions along a path.
  arXiv  Detail & Related papers  (2023-05-26T14:41:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.