Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparities

• URL: http://arxiv.org/abs/2507.13019v2
• Date: Fri, 26 Sep 2025 07:50:31 GMT
• Title: Rethinking the Embodied Gap in Vision-and-Language Navigation: A Holistic Study of Physical and Visual Disparities
• Authors: Liuyi Wang, Xinyuan Xia, Hui Zhao, Hanqing Wang, Tai Wang, Yilun Chen, Chengju Liu, Qijun Chen, Jiangmiao Pang,
• Abstract summary: We introduce VLN-PE, a physically realistic VLN platform supporting humanoid, quadruped, and wheeled robots.<n>For the first time, we evaluate several ego-centric VLN methods in physical robotic settings across different technical pipelines.<n>Our results reveal significant performance degradation due to limited robot observation space, environmental lighting variations, and physical challenges like collisions and falls.
• Score: 56.68006021126077
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent Vision-and-Language Navigation (VLN) advancements are promising, but their idealized assumptions about robot movement and control fail to reflect physically embodied deployment challenges. To bridge this gap, we introduce VLN-PE, a physically realistic VLN platform supporting humanoid, quadruped, and wheeled robots. For the first time, we systematically evaluate several ego-centric VLN methods in physical robotic settings across different technical pipelines, including classification models for single-step discrete action prediction, a diffusion model for dense waypoint prediction, and a train-free, map-based large language model (LLM) integrated with path planning. Our results reveal significant performance degradation due to limited robot observation space, environmental lighting variations, and physical challenges like collisions and falls. This also exposes locomotion constraints for legged robots in complex environments. VLN-PE is highly extensible, allowing seamless integration of new scenes beyond MP3D, thereby enabling more comprehensive VLN evaluation. Despite the weak generalization of current models in physical deployment, VLN-PE provides a new pathway for improving cross-embodiment's overall adaptability. We hope our findings and tools inspire the community to rethink VLN limitations and advance robust, practical VLN models. The code is available at https://crystalsixone.github.io/vln_pe.github.io/.

Related papers

• VLNVerse: A Benchmark for Vision-Language Navigation with Versatile, Embodied, Realistic Simulation and Evaluation [61.82502719679122]
  We introduce VLNVerse, a benchmark for Versatile, Embodied, Realistic Simulation, and Evaluation.<n>VLNVerse redefines VLN as a scalable, full-stack embodied AI problem.<n>We propose a novel unified multi-task model capable of addressing all tasks within the benchmark.
  arXiv  Detail & Related papers  (2025-12-22T04:27:26Z)
• VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning [77.34267241692706]
  Vision-Language Navigation (VLN) is a core challenge in embodied AI, requiring agents to navigate real-world environments using natural language instructions.<n>We propose VLN-R1, an end-to-end framework that leverages Large Vision-Language Models (LVLM) to directly translate egocentric video streams into continuous navigation actions.
  arXiv  Detail & Related papers  (2025-06-20T17:59:59Z)
• Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation [90.00687889213991]
  Solving complex long-horizon robotic manipulation problems requires sophisticated high-level planning capabilities.<n>Vision-language models (VLMs) pretrained on Internet data could in principle offer a framework for tackling such problems.<n>In this paper, we introduce a novel test-time framework that enhancesVLMs' physical reasoning capabilities for multi-stage manipulation tasks.
  arXiv  Detail & Related papers  (2025-02-23T20:42:15Z)
• ReVLA: Reverting Visual Domain Limitation of Robotic Foundation Models [55.07988373824348]
  We study the visual generalization capabilities of three existing robotic foundation models.<n>Our study shows that the existing models do not exhibit robustness to visual out-of-domain scenarios.<n>We propose a gradual backbone reversal approach founded on model merging.
  arXiv  Detail & Related papers  (2024-09-23T17:47:59Z)
• Human-Aware Vision-and-Language Navigation: Bridging Simulation to Reality with Dynamic Human Interactions [69.9980759344628]
  Vision-and-Language Navigation (VLN) aims to develop embodied agents that navigate based on human instructions. We introduce Human-Aware Vision-and-Language Navigation (HA-VLN), extending traditional VLN by incorporating dynamic human activities. We present the Expert-Supervised Cross-Modal (VLN-CM) and Non-Expert-Supervised Decision Transformer (VLN-DT) agents, utilizing cross-modal fusion and diverse training strategies.
  arXiv  Detail & Related papers  (2024-06-27T15:01:42Z)
• Explore the Potential Performance of Vision-and-Language Navigation Model: a Snapshot Ensemble Method [6.349841849317769]
  Vision-and-Language Navigation (VLN) is a challenging task in the field of artificial intelligence. We provide a new perspective to improve VLN models.
  arXiv  Detail & Related papers  (2021-11-28T23:07:48Z)
• Airbert: In-domain Pretraining for Vision-and-Language Navigation [91.03849833486974]
  Vision-and-language navigation (VLN) aims to enable embodied agents to navigate in realistic environments using natural language instructions. Recent methods explore pretraining to improve generalization of VLN agents. We introduce BnB, a large-scale and diverse in-domain VLN dataset.
  arXiv  Detail & Related papers  (2021-08-20T10:58:09Z)
• Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling [65.99956848461915]
  Vision-and-Language Navigation (VLN) is a task where agents must decide how to move through a 3D environment to reach a goal.<n>One of the problems of the VLN task is data scarcity since it is difficult to collect enough navigation paths with human-annotated instructions for interactive environments.<n>We propose an adversarial-driven counterfactual reasoning model that can consider effective conditions instead of low-quality augmented data.
  arXiv  Detail & Related papers  (2019-11-17T18:02:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.