Grounded Vision-Language Navigation for UAVs with Open-Vocabulary Goal Understanding

• URL: http://arxiv.org/abs/2506.10756v1
• Date: Thu, 12 Jun 2025 14:40:50 GMT
• Title: Grounded Vision-Language Navigation for UAVs with Open-Vocabulary Goal Understanding
• Authors: Yuhang Zhang, Haosheng Yu, Jiaping Xiao, Mir Feroskhan,
• Abstract summary: Vision-and-language navigation (VLN) is a long-standing challenge in autonomous robotics, aiming to empower agents with the ability to follow human instructions while navigating complex environments.<n>We propose Vision-Language Fly (VLFly), a framework tailored for Unmanned Aerial Vehicles (UAVs) to execute language-guided flight.
• Score: 1.280979348722635
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-and-language navigation (VLN) is a long-standing challenge in autonomous robotics, aiming to empower agents with the ability to follow human instructions while navigating complex environments. Two key bottlenecks remain in this field: generalization to out-of-distribution environments and reliance on fixed discrete action spaces. To address these challenges, we propose Vision-Language Fly (VLFly), a framework tailored for Unmanned Aerial Vehicles (UAVs) to execute language-guided flight. Without the requirement for localization or active ranging sensors, VLFly outputs continuous velocity commands purely from egocentric observations captured by an onboard monocular camera. The VLFly integrates three modules: an instruction encoder based on a large language model (LLM) that reformulates high-level language into structured prompts, a goal retriever powered by a vision-language model (VLM) that matches these prompts to goal images via vision-language similarity, and a waypoint planner that generates executable trajectories for real-time UAV control. VLFly is evaluated across diverse simulation environments without additional fine-tuning and consistently outperforms all baselines. Moreover, real-world VLN tasks in indoor and outdoor environments under direct and indirect instructions demonstrate that VLFly achieves robust open-vocabulary goal understanding and generalized navigation capabilities, even in the presence of abstract language input.

Related papers

• UAV-ON: A Benchmark for Open-World Object Goal Navigation with Aerial Agents [5.414995940540323]
  UAV-ON is a benchmark for large-scale Object Goal Navigation (NavObject) by aerial agents in open-world environments.<n>It comprises 14 high-fidelity Unreal Engine environments with diverse semantic regions and complex spatial layouts.<n>It defines 1270 annotated target objects, each characterized by an instance-level instruction that encodes category, physical footprint, and visual descriptors.
  arXiv  Detail & Related papers  (2025-08-01T03:23:06Z)
• VLN-R1: Vision-Language Navigation via Reinforcement Fine-Tuning [77.34267241692706]
  Vision-Language Navigation (VLN) is a core challenge in embodied AI, requiring agents to navigate real-world environments using natural language instructions.<n>We propose VLN-R1, an end-to-end framework that leverages Large Vision-Language Models (LVLM) to directly translate egocentric video streams into continuous navigation actions.
  arXiv  Detail & Related papers  (2025-06-20T17:59:59Z)
• Hierarchical Instruction-aware Embodied Visual Tracking [35.73851196966425]
  User-Centric Embodied Visual Tracking (UC-EVT) presents a novel challenge for reinforcement learning-based models.<n>We propose textbf Instruction-aware Embodied Visual Tracking (HIEVT) agent, which bridges instruction comprehension and action generation using textitspatial goals as intermediaries.
  arXiv  Detail & Related papers  (2025-05-27T04:36:26Z)
• UAV-VLN: End-to-End Vision Language guided Navigation for UAVs [0.0]
  A core challenge in AI-guided autonomy is enabling agents to navigate realistically and effectively in previously unseen environments.<n>We propose UAV-VLN, a novel end-to-end Vision-Language Navigation framework for Unmanned Aerial Vehicles (UAVs)<n>Our system interprets free-form natural language instructions, grounds them into visual observations, and plans feasible aerial trajectories in diverse environments.
  arXiv  Detail & Related papers  (2025-04-30T08:40:47Z)
• OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction [95.6266030753644]
  Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions.<n>Existing approaches require fine-tuning pre-trained vision-language models (VLMs) as visual and language features are independently fed into downstream policies.<n>We propose OTTER, a novel VLA architecture that leverages existing alignments through explicit, text-aware visual feature extraction.
  arXiv  Detail & Related papers  (2025-03-05T18:44:48Z)
• UnitedVLN: Generalizable Gaussian Splatting for Continuous Vision-Language Navigation [71.97405667493477]
  We introduce a novel, generalizable 3DGS-based pre-training paradigm, called UnitedVLN.<n>It enables agents to better explore future environments by unitedly rendering high-fidelity 360 visual images and semantic features.<n>UnitedVLN outperforms state-of-the-art methods on existing VLN-CE benchmarks.
  arXiv  Detail & Related papers  (2024-11-25T02:44:59Z)
• Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology [38.2096731046639]
  Recent efforts in UAV vision-language navigation predominantly adopt ground-based VLN settings. We propose solutions from three perspectives: platform, benchmark, and methodology.
  arXiv  Detail & Related papers  (2024-10-09T17:29:01Z)
• Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs [95.8010627763483]
  Mobility VLA is a hierarchical Vision-Language-Action (VLA) navigation policy that combines the environment understanding and common sense reasoning power of long-context VLMs. We show that Mobility VLA has a high end-to-end success rates on previously unsolved multimodal instructions.
  arXiv  Detail & Related papers  (2024-07-10T15:49:07Z)
• Vision and Language Navigation in the Real World via Online Visual Language Mapping [18.769171505280127]
  Vision-and-language navigation (VLN) methods are mainly evaluated in simulation. We propose a novel framework to address the VLN task in the real world. We evaluate the proposed pipeline on an Interbotix LoCoBot WX250 in an unseen lab environment.
  arXiv  Detail & Related papers  (2023-10-16T20:44:09Z)
• LangNav: Language as a Perceptual Representation for Navigation [63.90602960822604]
  We explore the use of language as a perceptual representation for vision-and-language navigation (VLN) Our approach uses off-the-shelf vision systems for image captioning and object detection to convert an agent's egocentric panoramic view at each time step into natural language descriptions.
  arXiv  Detail & Related papers  (2023-10-11T20:52:30Z)
• LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action [76.71101507291473]
  We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories. We show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data.
  arXiv  Detail & Related papers  (2022-07-10T10:41:50Z)
• Think Global, Act Local: Dual-scale Graph Transformer for Vision-and-Language Navigation [87.03299519917019]
  We propose a dual-scale graph transformer (DUET) for joint long-term action planning and fine-grained cross-modal understanding. We build a topological map on-the-fly to enable efficient exploration in global action space. The proposed approach, DUET, significantly outperforms state-of-the-art methods on goal-oriented vision-and-language navigation benchmarks.
  arXiv  Detail & Related papers  (2022-02-23T19:06:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.