Exposing the Long-tail in Embodied Urban Navigation via Scalable Learning from In-the-Wild Videos

• URL: http://arxiv.org/abs/2608.16476v1
• Date: Mon, 17 Aug 2026 12:15:59 GMT
• Title: Exposing the Long-tail in Embodied Urban Navigation via Scalable Learning from In-the-Wild Videos
• Abstract summary: We present a framework for learning point-goal urban navigation from web-scale in-the-wild egocentric videos.<n>The framework automatically annotates uncurated web videos with metric trajectories and structured navigation semantics.<n> Experiments on web-video data and real-world urban navigation tasks demonstrate effective knowledge transfer from unconstrained videos.
• Score: 5.797135182843089
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning embodied urban navigation policies from real-world data is constrained by the cost of task-specific data collection and the limited coverage of rare yet safety-critical scenarios. To address these challenges, we present a scalable framework for learning point-goal urban navigation from web-scale in-the-wild egocentric videos while systematically exposing its long tail. The framework automatically annotates uncurated web videos with metric trajectories and structured navigation semantics, which are then used to train a vision-language-action policy for interpretable navigation planning. We characterize the long tail based on model performance and the distribution of perception-motion patterns, and employ reflection-based analysis to diagnose recurring failure modes. Experiments on web-video data and real-world urban navigation tasks demonstrate effective knowledge transfer from unconstrained videos and reveal coherent long-tail structures beyond aggregate navigation performance.

Related papers

• AllDayNav: Lifelong Navigation via Real-World Reinforcement Learning [15.951251568937074]
  We propose AllDayNav, a lifelong self-learning navigation framework that implicitly encodes scene dynamics into the billion-scale parameters of a large model via reinforcement learning.<n>Experiments show that AllDayNav achieves success rates approaching $100%$ and consistently surpasses strong map-based, VLM, and RL baselines in path efficiency and robustness.
  arXiv  Detail & Related papers  (2026-06-09T14:35:53Z)
• Implicit Geometry Representations for Vision-and-Language Navigation from Web Videos [87.15961946494629]
  We introduce a large-scale video-instruction framework derived from web-based room tour videos.<n>Unlike existing datasets, our framework integrates open-ended description-enriched trajectories and action-enriched trajectories reconstructed in 3D.<n>A key extension in this work is the incorporation of implicit geometry representations, which extract spatial cues directly from RGB frames without requiring fragile 3D reconstruction.
  arXiv  Detail & Related papers  (2026-03-10T06:47:38Z)
• NavDreamer: Video Models as Zero-Shot 3D Navigators [10.105345998118915]
  We propose a video-based framework for 3D navigation that leverages generative video models as a universal interface between language instructions and trajectories.<n>Our main hypothesis is that video's ability to encode information and physical dynamics, combined with internet-scale availability, enables strong zero-shot generalization in navigation.
  arXiv  Detail & Related papers  (2026-02-10T13:24:12Z)
• From Seeing to Experiencing: Scaling Navigation Foundation Models with Reinforcement Learning [59.88543114325153]
  We introduce the Seeing-to-Experiencing framework to scale the capability of navigation foundation models with reinforcement learning.<n>S2E combines the strengths of pre-training on videos and post-training through RL.<n>We establish a comprehensive end-to-end evaluation benchmark, NavBench-GS, built on photorealistic 3DGS reconstructions of real-world scenes.
  arXiv  Detail & Related papers  (2025-07-29T17:26:10Z)
• VideoExplorer: Think With Videos For Agentic Long-Video Understanding [117.68219930263153]
  Long-video understanding is a challenging problem in computer vision.<n>We propose VideoExplorer, a framework grounded in the principle of thinking with video''<n>Rather than reasoning over a static context, VideoExplorer iteratively formulates sub-questions, locates relevant moments, and performs task-oriented, temporally scalable video understanding.
  arXiv  Detail & Related papers  (2025-06-12T15:39:10Z)
• NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistants [24.689242976554482]
  Navigating unfamiliar environments presents significant challenges for household robots.<n>Existing reinforcement learning methods cannot be directly transferred to new environments.<n>We try to transfer the logical knowledge and the generalization ability of pre-trained foundation models to zero-shot navigation.
  arXiv  Detail & Related papers  (2025-02-19T17:27:47Z)
• CityWalker: Learning Embodied Urban Navigation from Web-Scale Videos [11.912608309403359]
  We propose a scalable, data-driven approach for human-like urban navigation.<n>We train agents on thousands of hours of in-the-wild city walking and driving videos sourced from the web.<n>Our model learns sophisticated navigation policies to handle diverse challenges and critical scenarios.
  arXiv  Detail & Related papers  (2024-11-26T19:02:20Z)
• How To Not Train Your Dragon: Training-free Embodied Object Goal Navigation with Semantic Frontiers [94.46825166907831]
  We present a training-free solution to tackle the object goal navigation problem in Embodied AI. Our method builds a structured scene representation based on the classic visual simultaneous localization and mapping (V-SLAM) framework. Our method propagates semantics on the scene graphs based on language priors and scene statistics to introduce semantic knowledge to the geometric frontiers.
  arXiv  Detail & Related papers  (2023-05-26T13:38:33Z)
• Augmented reality navigation system for visual prosthesis [67.09251544230744]
  We propose an augmented reality navigation system for visual prosthesis that incorporates a software of reactive navigation and path planning. It consists on four steps: locating the subject on a map, planning the subject trajectory, showing it to the subject and re-planning without obstacles. Results show how our augmented navigation system help navigation performance by reducing the time and distance to reach the goals, even significantly reducing the number of obstacles collisions.
  arXiv  Detail & Related papers  (2021-09-30T09:41:40Z)
• Deep Learning for Embodied Vision Navigation: A Survey [108.13766213265069]
  "Embodied visual navigation" problem requires an agent to navigate in a 3D environment mainly rely on its first-person observation. This paper attempts to establish an outline of the current works in the field of embodied visual navigation by providing a comprehensive literature survey.
  arXiv  Detail & Related papers  (2021-07-07T12:09:04Z)
• Structured Scene Memory for Vision-Language Navigation [155.63025602722712]
  We propose a crucial architecture for vision-language navigation (VLN) It is compartmentalized enough to accurately memorize the percepts during navigation. It also serves as a structured scene representation, which captures and disentangles visual and geometric cues in the environment.
  arXiv  Detail & Related papers  (2021-03-05T03:41:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.