Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environments

• URL: http://arxiv.org/abs/2502.19024v1
• Date: Wed, 26 Feb 2025 10:30:40 GMT
• Title: Ground-level Viewpoint Vision-and-Language Navigation in Continuous Environments
• Authors: Zerui Li, Gengze Zhou, Haodong Hong, Yanyan Shao, Wenqi Lyu, Yanyuan Qiao, Qi Wu,
• Abstract summary: Vision-and-Language Navigation (VLN) agents associate time-sequenced visual observations with corresponding instructions to make decisions.<n>In this paper, we address the mismatch between human-centric instructions and quadruped robots with a low-height field of view.<n>We propose a Ground-level Viewpoint Navigation (GVNav) approach to mitigate this issue.
• Score: 10.953629652228024
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-and-Language Navigation (VLN) empowers agents to associate time-sequenced visual observations with corresponding instructions to make sequential decisions. However, generalization remains a persistent challenge, particularly when dealing with visually diverse scenes or transitioning from simulated environments to real-world deployment. In this paper, we address the mismatch between human-centric instructions and quadruped robots with a low-height field of view, proposing a Ground-level Viewpoint Navigation (GVNav) approach to mitigate this issue. This work represents the first attempt to highlight the generalization gap in VLN across varying heights of visual observation in realistic robot deployments. Our approach leverages weighted historical observations as enriched spatiotemporal contexts for instruction following, effectively managing feature collisions within cells by assigning appropriate weights to identical features across different viewpoints. This enables low-height robots to overcome challenges such as visual obstructions and perceptual mismatches. Additionally, we transfer the connectivity graph from the HM3D and Gibson datasets as an extra resource to enhance spatial priors and a more comprehensive representation of real-world scenarios, leading to improved performance and generalizability of the waypoint predictor in real-world environments. Extensive experiments demonstrate that our Ground-level Viewpoint Navigation (GVnav) approach significantly improves performance in both simulated environments and real-world deployments with quadruped robots.

Related papers

• Unseen from Seen: Rewriting Observation-Instruction Using Foundation Models for Augmenting Vision-Language Navigation [67.31811007549489]
  We propose a Rewriting-driven AugMentation (RAM) paradigm for Vision-Language Navigation (VLN) Benefiting from our rewriting mechanism, new observation-instruction can be obtained in both simulator-free and labor-saving manners to promote generalization. Experiments on both the discrete environments (R2R, REVERIE, and R4R) and continuous environments (R2R-CE) show the superior performance and impressive generalization ability of our method.
  arXiv  Detail & Related papers  (2025-03-23T13:18:17Z)
• World-Consistent Data Generation for Vision-and-Language Navigation [52.08816337783936]
  Vision-and-Language Navigation (VLN) is a challenging task that requires an agent to navigate through photorealistic environments following natural-language instructions.<n>One main obstacle existing in VLN is data scarcity, leading to poor generalization performance over unseen environments.<n>We propose the world-consistent data generation (WCGEN), an efficacious data-augmentation framework satisfying both diversity and world-consistency.
  arXiv  Detail & Related papers  (2024-12-09T11:40:54Z)
• AdaVLN: Towards Visual Language Navigation in Continuous Indoor Environments with Moving Humans [2.940962519388297]
  We propose an extension to the task, termed Adaptive Visual Language Navigation (AdaVLN)<n>AdaVLN requires robots to navigate complex 3D indoor environments populated with dynamically moving human obstacles.<n>We evaluate several baseline models on this task, analyze the unique challenges introduced by AdaVLN, and demonstrate its potential to bridge the sim-to-real gap in VLN research.
  arXiv  Detail & Related papers  (2024-11-27T17:36:08Z)
• UnitedVLN: Generalizable Gaussian Splatting for Continuous Vision-Language Navigation [71.97405667493477]
  We introduce a novel, generalizable 3DGS-based pre-training paradigm, called UnitedVLN. It enables agents to better explore future environments by unitedly rendering high-fidelity 360 visual images and semantic features. UnitedVLN outperforms state-of-the-art methods on existing VLN-CE benchmarks.
  arXiv  Detail & Related papers  (2024-11-25T02:44:59Z)
• Narrowing the Gap between Vision and Action in Navigation [28.753809306008996]
  We introduce a low-level action decoder jointly trained with high-level action prediction. Our agent can improve navigation performance metrics compared to the strong baselines on both high-level and low-level actions.
  arXiv  Detail & Related papers  (2024-08-19T20:09:56Z)
• AdaGlimpse: Active Visual Exploration with Arbitrary Glimpse Position and Scale [2.462953128215088]
  Active Visual Exploration (AVE) is a task that involves dynamically selecting observations (glimpses) Existing mobile platforms equipped with optical zoom capabilities can capture glimpses of arbitrary positions and scales. AdaGlimpse uses Soft Actor-Critic, a reinforcement learning algorithm tailored for exploration tasks, to select glimpses of arbitrary position and scale.
  arXiv  Detail & Related papers  (2024-04-04T14:35:49Z)
• CorNav: Autonomous Agent with Self-Corrected Planning for Zero-Shot Vision-and-Language Navigation [73.78984332354636]
  CorNav is a novel zero-shot framework for vision-and-language navigation. It incorporates environmental feedback for refining future plans and adjusting its actions. It consistently outperforms all baselines in a zero-shot multi-task setting.
  arXiv  Detail & Related papers  (2023-06-17T11:44:04Z)
• Towards Scale Consistent Monocular Visual Odometry by Learning from the Virtual World [83.36195426897768]
  We propose VRVO, a novel framework for retrieving the absolute scale from virtual data. We first train a scale-aware disparity network using both monocular real images and stereo virtual data. The resulting scale-consistent disparities are then integrated with a direct VO system.
  arXiv  Detail & Related papers  (2022-03-11T01:51:54Z)
• Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation [87.03299519917019]
  We propose a dual-scale graph transformer (DUET) for joint long-term action planning and fine-grained cross-modal understanding. We build a topological map on-the-fly to enable efficient exploration in global action space. The proposed approach, DUET, significantly outperforms state-of-the-art methods on goal-oriented vision-and-language navigation benchmarks.
  arXiv  Detail & Related papers  (2022-02-23T19:06:53Z)
• Vision-Language Navigation with Random Environmental Mixup [112.94609558723518]
  Vision-language Navigation (VLN) tasks require an agent to navigate step-by-step while perceiving the visual observations and comprehending a natural language instruction. Previous works have proposed various data augmentation methods to reduce data bias. We propose the Random Environmental Mixup (REM) method, which generates cross-connected house scenes as augmented data via mixuping environment.
  arXiv  Detail & Related papers  (2021-06-15T04:34:26Z)
• Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling [65.99956848461915]
  Vision-and-Language Navigation (VLN) is a task where agents must decide how to move through a 3D environment to reach a goal. One of the problems of the VLN task is data scarcity since it is difficult to collect enough navigation paths with human-annotated instructions for interactive environments. We propose an adversarial-driven counterfactual reasoning model that can consider effective conditions instead of low-quality augmented data.
  arXiv  Detail & Related papers  (2019-11-17T18:02:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.