Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos

• URL: http://arxiv.org/abs/2603.09259v1
• Date: Tue, 10 Mar 2026 06:47:38 GMT
• Title: Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos
• Abstract summary: We introduce a large-scale video-instruction framework derived from web-based room tour videos.<n>Unlike existing datasets, our framework integrates open-ended description-enriched trajectories and action-enriched trajectories reconstructed in 3D.<n>A key extension in this work is the incorporation of implicit geometry representations, which extract spatial cues directly from RGB frames without requiring fragile 3D reconstruction.
• Score: 87.15961946494629
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Vision-and-Language Navigation (VLN) has long been constrained by the limited diversity and scalability of simulator-curated datasets, which fail to capture the complexity of real-world environments. To overcome this limitation, we introduce a large-scale video-instruction framework derived from web-based room tour videos, enabling agents to learn from natural human walking demonstrations in diverse, realistic indoor settings. Unlike existing datasets, our framework integrates both open-ended description-enriched trajectories and action-enriched trajectories reconstructed in 3D, providing richer spatial and semantic supervision. A key extension in this work is the incorporation of implicit geometry representations, which extract spatial cues directly from RGB frames without requiring fragile 3D reconstruction. This approach substantially improves data utilization, alleviates reconstruction failures, and unlocks large portions of previously unusable video data. Comprehensive experiments across multiple VLN benchmarks (CVDN, SOON, R2R, and REVERIE) demonstrate that our method not only sets new state-of-the-art performance but also enables the development of robust zero-shot navigation agents. By bridging large-scale web videos with implicit spatial reasoning, this work advances embodied navigation towards more scalable, generalizable, and real-world applicable solutions.

Related papers

• Assisted Spatial Cognition Through Vision-Language Models [0.0]
  Multimodal AI is transforming assistive technologies by enabling simultaneous processing of visual and textual data.<n>This advancement holds significant promise for over 43 million visually impaired and neuro-divergent individuals worldwide.<n>We introduce a novel end-to-end framework that integrates Large Language Models (LLMs) and Vision-Language Models (VLMs)<n>Our system captures video input via standard mobile phone cameras, and employs SLAM3R to generate dense 3D point clouds in real-time.
  arXiv  Detail & Related papers  (2026-09-11T11:58:59Z)
• GA-VLN: Geometry-Aware BEV Representation for Efficient Vision-Language Navigation [75.85672467847631]
  We introduce the Geometry-Aware BEV (GA-BEV) - a compact, 3D-grounded feature representation.<n>We construct BEV spatial maps from RGB-D inputs by projecting visual features into 3D space.<n>We incorporate features from a pretrained 3D foundation model into the BEV space, injecting structural priors learned from large-scale 3D reconstruction tasks.
  arXiv  Detail & Related papers  (2026-05-21T06:20:17Z)
• RANGER: A Monocular Zero-Shot Semantic Navigation Framework through Contextual Adaptation [9.379574254353352]
  RANGER is a novel zero-shot, open-vocabulary semantic navigation framework that operates using only a monocular camera.<n>By simply observing a short video of a new environment, the system can also significantly improve task efficiency without requiring architectural modifications or fine-tuning.<n> Experiments on the HM3D benchmark and real-world environments demonstrate that RANGER achieves competitive performance in terms of navigation success rate and exploration efficiency.
  arXiv  Detail & Related papers  (2025-12-30T13:25:22Z)
• SpatialMosaic: A Multiview VLM Dataset for Partial Visibility [25.874299974251965]
  We propose a scalable multi-view data generation and annotation pipeline that constructs realistic spatial reasoning QAs.<n>We introduce SpatialMosaic-Bench, a benchmark for evaluating multi-view spatial reasoning under realistic and challenging scenarios.<n>We also present SpatialMosaicVLM, a hybrid framework that integrates 3D reconstruction models as geometry encoders within Vision-Language Models.
  arXiv  Detail & Related papers  (2025-12-29T10:48:54Z)
• Move to Understand a 3D Scene: Bridging Visual Grounding and Exploration for Efficient and Versatile Embodied Navigation [54.04601077224252]
  Embodied scene understanding requires not only comprehending visual-spatial information but also determining where to explore next in the 3D physical world.<n>underlinetextbf3D vision-language learning enables embodied agents to effectively explore and understand their environment.<n>model's versatility enables navigation using diverse input modalities, including categories, language descriptions, and reference images.
  arXiv  Detail & Related papers  (2025-07-05T14:15:52Z)
• Agentic 3D Scene Generation with Spatially Contextualized VLMs [67.31920821192323]
  We introduce a new paradigm that enables vision-language models to generate, understand, and edit complex 3D environments.<n>We develop an agentic 3D scene generation pipeline in which the VLM iteratively reads from and updates the spatial context.<n>Results show that our framework can handle diverse and challenging inputs, achieving a level of generalization not observed in prior work.
  arXiv  Detail & Related papers  (2025-05-26T15:28:17Z)
• SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observations [44.53106180688135]
  This work takes on the challenge of reconstructing 3D scenes from sparse or single-view inputs.<n>We introduce SpatialCrafter, a framework that leverages the rich knowledge in video diffusion models to generate plausible additional observations.<n>Through a trainable camera encoder and an epipolar attention mechanism for explicit geometric constraints, we achieve precise camera control and 3D consistency.
  arXiv  Detail & Related papers  (2025-05-17T13:05:13Z)
• Dynam3D: Dynamic Layered 3D Tokens Empower VLM for Vision-and-Language Navigation [61.21302433849139]
  Vision-and-Language Navigation (VLN) is a core task where embodied agents leverage their spatial mobility to navigate in 3D environments.<n>We propose Dynam3D, a dynamic layered 3D representation model that leverages language-aligned, generalizable, and hierarchical 3D representations as visual input to train 3D-VLM in navigation action prediction.<n>Our Dynam3D is capable of online encoding and localization of 3D instances, and dynamically updates them in changing environments to provide large-scale exploration and long-term memory capabilities for navigation.
  arXiv  Detail & Related papers  (2025-05-16T15:46:27Z)
• Volumetric Environment Representation for Vision-Language Navigation [66.04379819772764]
  Vision-language navigation (VLN) requires an agent to navigate through a 3D environment based on visual observations and natural language instructions. We introduce a Volumetric Environment Representation (VER), which voxelizes the physical world into structured 3D cells. VER predicts 3D occupancy, 3D room layout, and 3D bounding boxes jointly.
  arXiv  Detail & Related papers  (2024-03-21T06:14:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.