PGN: Design and Implementation of a Vision-Language Navigation System Based on Pangu Multimodal Foundation Model

• URL: http://arxiv.org/abs/2607.17806v1
• Date: Mon, 20 Jul 2026 10:48:48 GMT
• Title: PGN: Design and Implementation of a Vision-Language Navigation System Based on Pangu Multimodal Foundation Model
• Abstract summary: Vision-Language Navigation (VLN) requires an embodied agent to interpret a natural computation instruction and predict actions from temporally ordered visual observations.<n>This report presents PGN (Pangu Navigator), an offline VLN action-prediction system built on OpenPangu-7B.
• Score: 10.02560644098367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language Navigation (VLN) requires an embodied agent to interpret a natural-language instruction and predict actions from temporally ordered visual observations. Adapting a multimodal large language model to VLN requires visual-language alignment, compact temporal inputs, action-space grounding, and stable training on the target hardware. This technical report presents PGN (Pangu Navigator), an offline VLN action-prediction system built on OpenPangu-7B. Training proceeds in two stages. First, PGMM aligns a frozen EVA-ViT-G/14 vision encoder with the frozen language backbone by training a Q-Former and a two-layer MLP projector. Second, PGN adapts the aligned model to expert navigation trajectories using five-observation windows, epoch-dependent temporal sampling, and a reasoning-then-action output format; this stage freezes the aligned visual pathway and updates three structural-token embeddings and LoRA adapters. The implementation combines mixed-precision computation, selective FP32 computation, and DeepSpeed ZeRO-2 on eight Ascend 910B NPUs. Under teacher-forced, open-loop evaluation on 500 held-out expert trajectories, V9 reports a 62.29% Normalized Action Match (NAM) and a 100.00% Non-empty Rate (NER). These metrics quantify offline expert-action alignment rather than closed-loop navigation success; evaluating error accumulation, path efficiency, and goal completion remains future work.

Related papers

• FSD-VLN: Fast-Slow Dual-System Modeling for Aerial Long-Horizon Vision-Language Navigation [33.92221208368759]
  Vision-Language Navigation (VLN) is an autonomous navigation in unknown environments by mapping language instructions to real-time visual inputs.<n>Existing methods suffer from structural misalignment between global multimodal understanding and sequential action generation.<n>We propose FSD-VLN, a fast-slow dual-system architecture disentangling semantic reasoning and low-altitude flight command generation.
  arXiv  Detail & Related papers  (2026-07-09T11:17:04Z)
• LLaVA-OneVision-2: Towards Next-Generation Perceptual Intelligence [107.63317552620231]
  We present LLaVA-OneVision-2 (LLaVA-OV-2), the most capable vision-OneVision model to date.<n>It achieves superior performance across a broad range of multimodal benchmarks.<n>A standout capability is its unified perception across video understanding, temporal grounding, spatial grounding, and manipulation-language reasoning.
  arXiv  Detail & Related papers  (2026-05-25T15:54:04Z)
• AwareVLN: Reasoning with Self-awareness for Vision-Language Navigation [78.80536515102305]
  Vision-and-Language Navigation (VLN) requires an agent to ground language instructions to its own movement within a visual environment.<n>We propose AwareVLN, a novel framework that equips the navigation model with a self-aware reasoning mechanism.<n>Our approach features two key innovations: (1) a structural reasoning module that fosters spatial and task-oriented self-awareness, and (2) an automatic data engine with progress division for effective training.
  arXiv  Detail & Related papers  (2026-05-21T17:58:26Z)
• VP-VLA: Visual Prompting as an Interface for Vision-Language-Action Models [80.1055544841585]
  Vision-Language-Action models typically map visual observations and linguistic instructions directly to robotic control signals.<n>We propose VP-VLA, a dual-system framework that decouples high-level reasoning and low-level execution via a structured visual prompting interface.<n> Experiments on the Robocasa-GR1-Tabletop benchmark and SimplerEnv simulation demonstrate that VP-VLA improves success rates by 5% and 8.3%.
  arXiv  Detail & Related papers  (2026-03-23T14:08:58Z)
• AgentVLN: Towards Agentic Vision-and-Language Navigation [78.739525400071]
  Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments.<n>In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms.
  arXiv  Detail & Related papers  (2026-03-18T12:43:47Z)
• CoFL: Continuous Flow Fields for Language-Conditioned Navigation [14.35468089349405]
  CoFL is an end-to-end policy that maps a bird's-eye view (BEV) observation and a language instruction to a continuous flow field for navigation.<n>By training on a mixed distribution, CoFL significantly outperforms modular Vision-Language Model (VLM)-based planners.
  arXiv  Detail & Related papers  (2026-03-03T11:02:55Z)
• AutoFly: Vision-Language-Action Model for UAV Autonomous Navigation in the Wild [62.47761809929869]
  Vision-language navigation (VLN) requires intelligent agents to navigate environments by interpreting linguistic instructions alongside visual observations.<n>Current VLN research for unmanned aerial vehicles (UAVs) relies on detailed, pre-specified instructions to guide the UAV along predetermined routes.<n>We propose AutoFly, an end-to-end Vision-Language-Action model for autonomous UAV navigation.
  arXiv  Detail & Related papers  (2026-02-10T11:08:07Z)
• SeeNav-Agent: Enhancing Vision-Language Navigation with Visual Prompt and Step-Level Policy Optimization [32.785475974900244]
  Existing Vision-Language Navigation (VLN) agents often suffer from perception errors, reasoning errors, and planning errors.<n>To address these limitations, a novel VLN agent framework, named SeeNav-Agent, is proposed in this work.<n>Step Reward Group Policy Optimization (SRGPO) is designed for the post-training of VLN agents.
  arXiv  Detail & Related papers  (2025-12-02T10:40:46Z)
• VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning [77.34267241692706]
  Vision-Language Navigation (VLN) is a core challenge in embodied AI, requiring agents to navigate real-world environments using natural language instructions.<n>We propose VLN-R1, an end-to-end framework that leverages Large Vision-Language Models (LVLM) to directly translate egocentric video streams into continuous navigation actions.
  arXiv  Detail & Related papers  (2025-06-20T17:59:59Z)
• Reinforced Structured State-Evolution for Vision-Language Navigation [42.46176089721314]
  Vision-and-language Navigation (VLN) task requires an embodied agent to navigate to a remote location following a natural language instruction. Previous methods usually adopt a sequence model (e.g., Transformer and LSTM) as the navigator. We propose a novel Structured state-Evolution (SEvol) model to effectively maintain the environment layout clues for VLN.
  arXiv  Detail & Related papers  (2022-04-20T07:51:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.