Aerial Vision-and-Language Navigation with Grid-based View Selection and Map Construction

• URL: http://arxiv.org/abs/2503.11091v1
• Date: Fri, 14 Mar 2025 05:20:43 GMT
• Title: Aerial Vision-and-Language Navigation with Grid-based View Selection and Map Construction
• Authors: Ganlong Zhao, Guanbin Li, Jia Pan, Yizhou Yu,
• Abstract summary: Aerial Vision-and-Language Navigation (Aerial VLN) aims to obtain an unmanned aerial vehicle agent to navigate aerial 3D environments following human instruction.<n>Previous methods struggle to perform well due to the longer navigation path, more complicated 3D scenes, and the neglect of the interplay between vertical and horizontal actions.<n>We propose a novel grid-based view selection framework that formulates aerial VLN action prediction as a grid-based view selection task.
• Score: 102.70482302750897
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Aerial Vision-and-Language Navigation (Aerial VLN) aims to obtain an unmanned aerial vehicle agent to navigate aerial 3D environments following human instruction. Compared to ground-based VLN, aerial VLN requires the agent to decide the next action in both horizontal and vertical directions based on the first-person view observations. Previous methods struggle to perform well due to the longer navigation path, more complicated 3D scenes, and the neglect of the interplay between vertical and horizontal actions. In this paper, we propose a novel grid-based view selection framework that formulates aerial VLN action prediction as a grid-based view selection task, incorporating vertical action prediction in a manner that accounts for the coupling with horizontal actions, thereby enabling effective altitude adjustments. We further introduce a grid-based bird's eye view map for aerial space to fuse the visual information in the navigation history, provide contextual scene information, and mitigate the impact of obstacles. Finally, a cross-modal transformer is adopted to explicitly align the long navigation history with the instruction. We demonstrate the superiority of our method in extensive experiments.

Related papers

• Beyond the Horizon: Decoupling UAVs Multi-View Action Recognition via Partial Order Transfer [38.646757044416866]
  We introduce a multi-view formulation tailored to varying UAV altitudes and empirically observe a partial order among views. This motivates a novel approach that explicitly models the hierarchical structure of UAV views to improve recognition performance across altitudes. We propose the Partial Order Guided Multi-View Network (POG-MVNet), designed to address drastic view variations.
  arXiv  Detail & Related papers  (2025-04-29T08:22:13Z)
• UnitedVLN: Generalizable Gaussian Splatting for Continuous Vision-Language Navigation [71.97405667493477]
  We introduce a novel, generalizable 3DGS-based pre-training paradigm, called UnitedVLN. It enables agents to better explore future environments by unitedly rendering high-fidelity 360 visual images and semantic features. UnitedVLN outperforms state-of-the-art methods on existing VLN-CE benchmarks.
  arXiv  Detail & Related papers  (2024-11-25T02:44:59Z)
• Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology [38.2096731046639]
  Recent efforts in UAV vision-language navigation predominantly adopt ground-based VLN settings. We propose solutions from three perspectives: platform, benchmark, and methodology.
  arXiv  Detail & Related papers  (2024-10-09T17:29:01Z)
• CityNav: Language-Goal Aerial Navigation Dataset with Geographic Information [25.51740922661166]
  Vision-and-language navigation (VLN) aims to guide autonomous agents through real-world environments by integrating visual and linguistic cues. We introduce CityNav, a novel dataset explicitly designed for language-guided aerial navigation in 3D environments of real cities. CityNav comprises 32k natural language descriptions paired with human demonstration trajectories, collected via a newly developed web-based 3D simulator.
  arXiv  Detail & Related papers  (2024-06-20T12:08:27Z)
• Pixel to Elevation: Learning to Predict Elevation Maps at Long Range using Images for Autonomous Offroad Navigation [10.898724668444125]
  We present a learning-based approach capable of predicting terrain elevation maps at long-range using only onboard egocentric images in real-time. We experimentally validate the applicability of our proposed approach for autonomous offroad robotic navigation in complex and unstructured terrain.
  arXiv  Detail & Related papers  (2024-01-30T22:37:24Z)
• Learning Navigational Visual Representations with Semantic Map Supervision [85.91625020847358]
  We propose a navigational-specific visual representation learning method by contrasting the agent's egocentric views and semantic maps. Ego$2$-Map learning transfers the compact and rich information from a map, such as objects, structure and transition, to the agent's egocentric representations for navigation.
  arXiv  Detail & Related papers  (2023-07-23T14:01:05Z)
• ESceme: Vision-and-Language Navigation with Episodic Scene Memory [72.69189330588539]
  Vision-and-language navigation (VLN) simulates a visual agent that follows natural-language navigation instructions in real-world scenes. We introduce a mechanism of Episodic Scene memory (ESceme) for VLN that wakes an agent's memories of past visits when it enters the current scene.
  arXiv  Detail & Related papers  (2023-03-02T07:42:07Z)
• Tri-Perspective View for Vision-Based 3D Semantic Occupancy Prediction [84.94140661523956]
  We propose a tri-perspective view (TPV) representation which accompanies BEV with two additional perpendicular planes. We model each point in the 3D space by summing its projected features on the three planes. Experiments show that our model trained with sparse supervision effectively predicts the semantic occupancy for all voxels.
  arXiv  Detail & Related papers  (2023-02-15T17:58:10Z)
• VPAIR -- Aerial Visual Place Recognition and Localization in Large-scale Outdoor Environments [49.82314641876602]
  We present a new dataset named VPAIR. The dataset was recorded on board a light aircraft flying at an altitude of more than 300 meters above ground. The dataset covers a more than one hundred kilometers long trajectory over various types of challenging landscapes.
  arXiv  Detail & Related papers  (2022-05-23T18:50:08Z)
• History Aware Multimodal Transformer for Vision-and-Language Navigation [96.80655332881432]
  Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes. We introduce a History Aware Multimodal Transformer (HAMT) to incorporate a long-horizon history into multimodal decision making.
  arXiv  Detail & Related papers  (2021-10-25T22:54:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.