FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation

• URL: http://arxiv.org/abs/2606.30367v1
• Date: Mon, 29 Jun 2026 14:33:03 GMT
• Title: FutureNav: Unified World-Action Modeling for Vision-and-Language Navigation
• Abstract summary: FutureNav is a VLM-based unified world-action modeling framework for vision-and-language navigation.<n>We show that FutureNav achieves state-of-the-art performance on multiple VLN benchmarks and substantially outperforms prior VLN methods.
• Score: 22.882723669402026
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-and-language navigation (VLN) in continuous environments requires an agent to ground instructions in egocentric observations while maintaining spatial understanding across long action sequences. Recent navigation foundation models have shown strong progress by scaling vision-language models, but they often learn navigation primarily as direct action generation, without explicitly modeling world states or predicting their future evolution. We introduce FutureNav, a VLM-based unified world-action modeling framework for vision-and-language navigation. Specifically, FutureNav jointly encodes text, visual, and spatial features and feeds them into the LLM, and optimizes four objectives for simultaneous world and action modeling: an action policy objective for navigation action prediction, inverse and forward dynamics objectives for modeling state transitions, and a future generation objective for predicting future spatial states. This unified architecture strengthens action prediction while explicitly modeling the world, without sacrificing inference speed. Extensive experiments show that, with only a 4B-scale backbone, FutureNav achieves state-of-the-art performance on multiple VLN benchmarks and substantially outperforms prior VLN methods, paving the way toward future world-action models for VLN. We will release the code and models to support future research.

Related papers

• WNM-3D: A World Navigation Model with 3D Scene Conditioning for Closed-Loop VLN [53.419929910272735]
  We present WNM-3D, a generative World Navigation Model with 3D scene conditioning for continuous VLN.<n>To consolidate past observations into persistent scene context, a frozen feed-forward geometry encoder extracts geometry-aware representations from the monocular egocentric RGB history.<n>Through block-causal attention, this prefix conditions every future video-action block, providing a shared geometric context for both future-view and action generation.
  arXiv  Detail & Related papers  (2026-08-07T14:29:00Z)
• NavWAM: A Navigation World Action Model for Goal-Conditioned Visual Navigation [41.592946383221566]
  navigation world models provide such visual foresight, but they remain prediction modules that require an external planner to convert predicted futures into closed-loop control.<n>We propose Navigation World Action Model (NavWAM), a diffusion-transformer policy that turns navigation world-model prediction into executable action.<n>We build NavWAM through simulation pretraining and real-robot adaptation, and evaluate it on image-goal navigation against planning-based world models and a representative direct navigation policy.
  arXiv  Detail & Related papers  (2026-06-11T15:44:36Z)
• Learning Vision-Language-Action World Models for Autonomous Driving [15.103497388527943]
  Vision-Language-Action (VLA) models have recently achieved notable progress in end-to-end autonomous driving.<n>VLA models often lack explicit modeling of temporal dynamics and global world consistency.<n>We present VLA-World, a simple yet effective VLA world model that unifies predictive imagination with reflective reasoning.
  arXiv  Detail & Related papers  (2026-04-10T07:38:05Z)
• WorldMAP: Bootstrapping Vision-Language Navigation Trajectory Prediction with Generative World Models [32.71152441572375]
  Vision-language models (VLMs) and generative world models are opening new opportunities for embodied navigation.<n>We present WorldMAP, a framework that converts world-model-generated futures into persistent semantic-spatial structure and planning-derived supervision.
  arXiv  Detail & Related papers  (2026-04-09T08:21:58Z)
• DriveDreamer-Policy: A Geometry-Grounded World-Action Model for Unified Generation and Planning [44.543763428623976]
  We present DriveDreamer-Policy, a unified driving world-action model that integrates depth generation, future video generation, and motion planning.<n>The proposed model produces more coherent imagined futures and more informed driving actions, while maintaining modularity and latency controllable.
  arXiv  Detail & Related papers  (2026-04-02T08:33:18Z)
• Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving [52.04950569530877]
  We present Uni-World VLA, a unified vision-language-action model that tightly interleaves future frame prediction and trajectory planning.<n>Our approach achieves competitive closed-loop planning performance while producing high-fidelity future frame predictions.
  arXiv  Detail & Related papers  (2026-03-28T14:39:51Z)
• Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation [66.7879424097418]
  We present WorldDrive, a holistic framework that couples scene generation and real-time planning via unifying vision and motion representation.<n>A simple interaction between motion representation, visual representation, and ego status can generate high-quality, multi-modal trajectories.<n>Experiments on the NAVSIM, NAVSIM-v2, and nuScenes benchmarks demonstrate that WorldDrive achieves leading planning performance among vision-only methods.
  arXiv  Detail & Related papers  (2026-03-16T07:59:39Z)
• Aerial World Model for Long-horizon Visual Generation and Navigation in 3D Space [48.19308247102762]
  We propose ANWM, an aerial navigation world model that predicts future visual observations conditioned on past frames and actions.<n> ANWM is trained on 4-DoF UAV trajectories and introduces a physics-inspired module: Future Frame Projection.<n> Empirical results demonstrate that ANWM significantly outperforms existing world models in long-distance visual forecasting and improves UAV navigation success rates in large-scale environments.
  arXiv  Detail & Related papers  (2025-12-26T06:22:39Z)
• Can World Models Benefit VLMs for World Dynamics? [59.73433292793044]
  We investigate the capabilities when world model priors are transferred into Vision-Language Models.<n>We name our best-performing variant Dynamic Vision Aligner (DyVA)<n>We find DyVA to surpass both open-source and proprietary baselines, achieving state-of-the-art or comparable performance.
  arXiv  Detail & Related papers  (2025-10-01T13:07:05Z)
• Navigation World Models [68.58459393846461]
  We introduce a controllable video generation model that predicts future visual observations based on past observations and navigation actions.<n>In familiar environments, NWM can plan navigation trajectories by simulating them and evaluating whether they achieve the desired goal.<n>Experiments demonstrate its effectiveness in planning trajectories from scratch or by ranking trajectories sampled from an external policy.
  arXiv  Detail & Related papers  (2024-12-04T18:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.