Seeing What Matters: Visual Cue Guided Video Planning for Generalizable Robot Navigation

• URL: http://arxiv.org/abs/2609.16737v1
• Date: Tue, 15 Sep 2026 07:11:21 GMT
• Title: Seeing What Matters: Visual Cue Guided Video Planning for Generalizable Robot Navigation
• Abstract summary: We present CueNav, a model-based navigation framework combining visual cue guided video planning with an embodiment-specific Inverse-Dynamics Model (IDM)<n>With the visual cue encoding global task context, CueNav achieves nearly 2x higher success in maze navigation than planning without the cue.<n>Our results show that visual cue-guided video planning with embodiment-specific action grounding paves the way toward a generalizable navigation framework.
• Score: 12.951864480301294
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative video models can serve as a promising backbone for robot navigation by predicting future observations as video plans. Recent approaches often condition video planning on short-horizon guidance and recover geometric waypoints through scene reconstruction, leaving longer-horizon planning and precise video-to-action translation less explored. We present CueNav, a video model-based navigation framework combining visual cue guided video planning with an embodiment-specific Inverse-Dynamics Model (IDM). As visual cues, we use a Bird's-Eye View (BEV) map to convey global task context and retain part of the robot body in the egocentric observation to expose embodiment context. These cues guide the video planner, while the IDM translates dense flow fields extracted from the video plan into robot actions. With the visual cue encoding global task context, CueNav achieves nearly 2x higher success in maze navigation than planning without the cue. The body-aware view with the IDM enables precise navigation with 70% success in a narrow passage where comparison methods largely fail to complete the task. We further demonstrate zero-shot semantic-conditioned navigation and deployment of the same video planner across different robot platforms. Our results show that visual cue-guided video planning with embodiment-specific action grounding paves the way toward a generalizable navigation framework for longer-horizon planning and embodiment-aware control. Additional results and code are available on our project website: https://cuenav.github.io.

Related papers

• ImagiNav: Scalable Embodied Navigation via Generative Visual Prediction and Inverse Dynamics [13.382453086651019]
  Vision-Language Navigation has relied on end-to-end policies trained on expensive, embodiment-specific robot data.<n>We propose ImagiNav, a novel modular paradigm that decouples visual planning from robot actuation.<n>ImagiNav demonstrates strong zero-shot transfer to robot navigation without requiring robot demonstrations.
  arXiv  Detail & Related papers  (2026-03-14T08:34:02Z)
• NavDreamer: Video Models as Zero-Shot 3D Navigators [10.105345998118915]
  We propose a video-based framework for 3D navigation that leverages generative video models as a universal interface between language instructions and trajectories.<n>Our main hypothesis is that video's ability to encode information and physical dynamics, combined with internet-scale availability, enables strong zero-shot generalization in navigation.
  arXiv  Detail & Related papers  (2026-02-10T13:24:12Z)
• Grounding Generated Videos in Feasible Plans via World Models [52.63206803295352]
  Grounding Video Plans with World Models (GVP-WM) is a planning method that grounds video-generated plans into feasible action sequences.<n>GVP-WM first generates a video plan from initial and goal observations, then projects the video guidance onto the manifold of dynamically feasible latent trajectories.
  arXiv  Detail & Related papers  (2026-02-02T11:04:47Z)
• YOPO-Nav: Visual Navigation using 3DGS Graphs from One-Pass Videos [5.7772802828964664]
  We propose YOPO-Nav, which encodes an environment into a compact spatial representation composed of interconnected local 3D Gaussian Splatting (3DGS) models.<n>During navigation, the framework aligns the robot's current visual observation with this representation and predicts actions that guide it back toward the demonstrated trajectory.<n>We benchmark recent visual navigation methods on trajectories from YOPO-Campus using a Clearpath Jackal robot. Experimental results show YOPO-Nav provides excellent performance in image-goal navigation for real-world scenes on a physical robot.
  arXiv  Detail & Related papers  (2025-12-10T18:32:38Z)
• NavForesee: A Unified Vision-Language World Model for Hierarchical Planning and Dual-Horizon Navigation Prediction [12.352236127154761]
  We introduce NavForesee, a novel Vision-Language Model (VLM) that unifies high-level language planning and predictive world model imagination.<n>Our approach empowers a single VLM to concurrently perform planning and predictive foresight.<n>Our work highlights the immense potential of fusing explicit language planning with implicittemporal prediction, paving the way for more intelligent and capable embodied agents.
  arXiv  Detail & Related papers  (2025-12-01T11:24:16Z)
• Aerial Vision-and-Language Navigation with Grid-based View Selection and Map Construction [102.70482302750897]
  Aerial Vision-and-Language Navigation (Aerial VLN) aims to obtain an unmanned aerial vehicle agent to navigate aerial 3D environments following human instruction.<n>Previous methods struggle to perform well due to the longer navigation path, more complicated 3D scenes, and the neglect of the interplay between vertical and horizontal actions.<n>We propose a novel grid-based view selection framework that formulates aerial VLN action prediction as a grid-based view selection task.
  arXiv  Detail & Related papers  (2025-03-14T05:20:43Z)
• Navigation World Models [68.58459393846461]
  We introduce a controllable video generation model that predicts future visual observations based on past observations and navigation actions.<n>In familiar environments, NWM can plan navigation trajectories by simulating them and evaluating whether they achieve the desired goal.<n>Experiments demonstrate its effectiveness in planning trajectories from scratch or by ranking trajectories sampled from an external policy.
  arXiv  Detail & Related papers  (2024-12-04T18:59:45Z)
• ESceme: Vision-and-Language Navigation with Episodic Scene Memory [72.69189330588539]
  Vision-and-language navigation (VLN) simulates a visual agent that follows natural-language navigation instructions in real-world scenes. We introduce a mechanism of Episodic Scene memory (ESceme) for VLN that wakes an agent's memories of past visits when it enters the current scene.
  arXiv  Detail & Related papers  (2023-03-02T07:42:07Z)
• LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action [76.71101507291473]
  We present a system, LM-Nav, for robotic navigation that enjoys the benefits of training on unannotated large datasets of trajectories. We show that such a system can be constructed entirely out of pre-trained models for navigation (ViNG), image-language association (CLIP), and language modeling (GPT-3), without requiring any fine-tuning or language-annotated robot data.
  arXiv  Detail & Related papers  (2022-07-10T10:41:50Z)
• ViKiNG: Vision-Based Kilometer-Scale Navigation with Geographic Hints [94.60414567852536]
  Long-range navigation requires both planning and reasoning about local traversability. We propose a learning-based approach that integrates learning and planning. ViKiNG can leverage its image-based learned controller and goal-directed to navigate to goals up to 3 kilometers away.
  arXiv  Detail & Related papers  (2022-02-23T02:14:23Z)
• Structured Scene Memory for Vision-Language Navigation [155.63025602722712]
  We propose a crucial architecture for vision-language navigation (VLN) It is compartmentalized enough to accurately memorize the percepts during navigation. It also serves as a structured scene representation, which captures and disentangles visual and geometric cues in the environment.
  arXiv  Detail & Related papers  (2021-03-05T03:41:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.