OmniVLN: Omnidirectional 3D Perception and Token-Efficient LLM Reasoning for Visual-Language Navigation across Air and Ground Platforms

• URL: http://arxiv.org/abs/2603.17351v1
• Date: Wed, 18 Mar 2026 04:26:30 GMT
• Title: OmniVLN: Omnidirectional 3D Perception and Token-Efficient LLM Reasoning for Visual-Language Navigation across Air and Ground Platforms
• Abstract summary: Language-guided embodied navigation requires an agent to interpret object-referential instructions, search across multiple rooms, localize the referenced target, and execute reliable motion toward it.<n>We present OmniVLN, a zero-shot visual-language navigation framework that couples omnidirectional 3D perception with token-efficient hierarchical reasoning for both aerial and ground robots.<n> Experiments show that the proposed hierarchical interface improves spatial referring accuracy from 77.27% to 93.18%, reduces cumulative prompt tokens by up to 61.7% in cluttered multi-room settings, and improves navigation success by up to 11.68% over a flat-
• Score: 33.40889181799252
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language-guided embodied navigation requires an agent to interpret object-referential instructions, search across multiple rooms, localize the referenced target, and execute reliable motion toward it. Existing systems remain limited in real indoor environments because narrow field-of-view sensing exposes only a partial local scene at each step, often forcing repeated rotations, delaying target discovery, and producing fragmented spatial understanding; meanwhile, directly prompting LLMs with dense 3D maps or exhaustive object lists quickly exceeds the context budget. We present OmniVLN, a zero-shot visual-language navigation framework that couples omnidirectional 3D perception with token-efficient hierarchical reasoning for both aerial and ground robots. OmniVLN fuses a rotating LiDAR and panoramic vision into a hardware-agnostic mapping stack, incrementally constructs a five-layer Dynamic Scene Graph (DSG) from mesh geometry to room- and building-level structure, and stabilizes high-level topology through persistent-homology-based room partitioning and hybrid geometric/VLM relation verification. For navigation, the global DSG is transformed into an agent-centric 3D octant representation with multi-resolution spatial attention prompting, enabling the LLM to progressively filter candidate rooms, infer egocentric orientation, localize target objects, and emit executable navigation primitives while preserving fine local detail and compact long-range memory. Experiments show that the proposed hierarchical interface improves spatial referring accuracy from 77.27\% to 93.18\%, reduces cumulative prompt tokens by up to 61.7\% in cluttered multi-room settings, and improves navigation success by up to 11.68\% over a flat-list baseline. We will release the code and an omnidirectional multimodal dataset to support reproducible research.

Related papers

• Assisted Spatial Cognition Through Vision-Language Models [0.0]
  Multimodal AI is transforming assistive technologies by enabling simultaneous processing of visual and textual data.<n>This advancement holds significant promise for over 43 million visually impaired and neuro-divergent individuals worldwide.<n>We introduce a novel end-to-end framework that integrates Large Language Models (LLMs) and Vision-Language Models (VLMs)<n>Our system captures video input via standard mobile phone cameras, and employs SLAM3R to generate dense 3D point clouds in real-time.
  arXiv  Detail & Related papers  (2026-09-11T11:58:59Z)
• Instance-Enriched Semantic Maps for Visual Language Navigation [9.24074870738429]
  Visual Language Navigation aims to enable an embodied agent to navigate complex environments by following natural language instructions.<n>Existing systems lack instance-level object detail and robustness to diverse user queries, limiting reliable navigation in complex indoor environments.<n>We propose Instance-Enriched Semantic Maps, a unified framework with three key contributions.
  arXiv  Detail & Related papers  (2026-07-14T11:09:48Z)
• Bridging the 2D-3D Gap: A Hierarchical Semantic-Geometric Map for Vision Language Navigation [32.02017382315305]
  Vision-Language Navigation (VLN) enables embodied agents to reach target locations in unseen environments by following language instructions.<n>Despite recent progress with vision-language models (VLMs), a critical semantic-geometric gap remains.<n>We propose a Hierarchical Semantic-Geometric Map (HSGM) that transforms 3D geometric information into a structured representation compatible withVLMs.
  arXiv  Detail & Related papers  (2026-05-25T08:53:21Z)
• GoalVLM: VLM-driven Object Goal Navigation for Multi-Agent System [0.0]
  We present GoalVLM, a cooperative multi-agent framework for zero-shot, open-vocabulary object navigation.<n>GoalVLM integrates a Vision-Language Model (VLM) directly into the decision loop, SAM3 for text-prompted detection and segmentation, and SpaceOM for spatial reasoning.<n>We evaluate GoalVLM on GOAT-Bench val_unseen, where each episode requires navigating to a chain of 5-7 open-vocabulary targets.
  arXiv  Detail & Related papers  (2026-03-18T18:59:33Z)
• AgentVLN: Towards Agentic Vision-and-Language Navigation [78.739525400071]
  Vision-and-Language Navigation (VLN) requires an embodied agent to ground complex natural-language instructions into long-horizon navigation in unseen environments.<n>In this paper, we propose AgentVLN, a novel and efficient embodied navigation framework that can be deployed on edge computing platforms.
  arXiv  Detail & Related papers  (2026-03-18T12:43:47Z)
• 3DGSNav: Enhancing Vision-Language Model Reasoning for Object Navigation via Active 3D Gaussian Splatting [12.057873540714098]
  3DGSNav is a novel framework that embeds 3D Gaussian Splatting (3DGS) as persistent memory for vision-language models (VLMs) to enhance spatial reasoning.<n>3DGSNav incrementally constructs a 3DGS representation of the environment, enabling trajectory-guided free-viewpoint rendering of frontier-aware first-person views.<n>During navigation, a real-time object detector filters potential targets, while VLM-driven active viewpoint switching performs target re-verification.
  arXiv  Detail & Related papers  (2026-02-12T16:41:26Z)
• FOM-Nav: Frontier-Object Maps for Object Goal Navigation [65.76906445210112]
  FOM-Nav is a framework that enhances exploration efficiency through Frontier-Object Maps and vision-language models.<n>To train FOM-Nav, we automatically construct large-scale navigation datasets from real-world scanned environments.<n> FOM-Nav achieves state-of-the-art performance on the MP3D and HM3D benchmarks, particularly in navigation efficiency metric SPL.
  arXiv  Detail & Related papers  (2025-11-30T18:16:09Z)
• TopV-Nav: Unlocking the Top-View Spatial Reasoning Potential of MLLM for Zero-shot Object Navigation [52.422619828854984]
  We introduce TopV-Nav, an MLLM-based method that directly reasons on the top-view map with sufficient spatial information.<n>To fully unlock the MLLM's spatial reasoning potential in top-view perspective, we propose the Adaptive Visual Prompt Generation (AVPG) method.
  arXiv  Detail & Related papers  (2024-11-25T14:27:55Z)
• SG-Nav: Online 3D Scene Graph Prompting for LLM-based Zero-shot Object Navigation [83.4599149936183]
  Existing zero-shot object navigation methods prompt LLM with the text of spatially closed objects. We propose to represent the observed scene with 3D scene graph. We conduct extensive experiments on MP3D, HM3D and RoboTHOR environments, where SG-Nav surpasses previous state-of-the-art zero-shot methods by more than 10% SR on all benchmarks.
  arXiv  Detail & Related papers  (2024-10-10T17:57:19Z)
• MMScan: A Multi-Modal 3D Scene Dataset with Hierarchical Grounded Language Annotations [55.022519020409405]
  This paper builds the first largest ever multi-modal 3D scene dataset and benchmark with hierarchical grounded language annotations, MMScan.<n>The resulting multi-modal 3D dataset encompasses 1.4M meta-annotated captions on 109k objects and 7.7k regions as well as over 3.04M diverse samples for 3D visual grounding and question-answering benchmarks.
  arXiv  Detail & Related papers  (2024-06-13T17:59:30Z)
• Structured Scene Memory for Vision-Language Navigation [155.63025602722712]
  We propose a crucial architecture for vision-language navigation (VLN) It is compartmentalized enough to accurately memorize the percepts during navigation. It also serves as a structured scene representation, which captures and disentangles visual and geometric cues in the environment.
  arXiv  Detail & Related papers  (2021-03-05T03:41:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.