Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation

• URL: http://arxiv.org/abs/2603.14948v1
• Date: Mon, 16 Mar 2026 07:59:39 GMT
• Title: Bridging Scene Generation and Planning: Driving with World Model via Unifying Vision and Motion Representation
• Abstract summary: We present WorldDrive, a holistic framework that couples scene generation and real-time planning via unifying vision and motion representation.<n>A simple interaction between motion representation, visual representation, and ego status can generate high-quality, multi-modal trajectories.<n>Experiments on the NAVSIM, NAVSIM-v2, and nuScenes benchmarks demonstrate that WorldDrive achieves leading planning performance among vision-only methods.
• Score: 66.7879424097418
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: End-to-end autonomous driving aims to generate safe and plausible planning policies from raw sensor input. Driving world models have shown great potential in learning rich representations by predicting the future evolution of a driving scene. However, existing driving world models primarily focus on visual scene representation, and motion representation is not explicitly designed to be planner-shared and inheritable, leaving a schism between the optimization of visual scene generation and the requirements of precise motion planning. We present WorldDrive, a holistic framework that couples scene generation and real-time planning via unifying vision and motion representation. We first introduce a Trajectory-aware Driving World Model, which conditions on a trajectory vocabulary to enforce consistency between visual dynamics and motion intentions, enabling the generation of diverse and plausible future scenes conditioned on a specific trajectory. We transfer the vision and motion encoders to a downstream Multi-modal Planner, ensuring the driving policy operates on mature representations pre-optimized by scene generation. A simple interaction between motion representation, visual representation, and ego status can generate high-quality, multi-modal trajectories. Furthermore, to exploit the world model's foresight, we propose a Future-aware Rewarder, which distills future latent representation from the frozen world model to evaluate and select optimal trajectories in real-time. Extensive experiments on the NAVSIM, NAVSIM-v2, and nuScenes benchmarks demonstrate that WorldDrive achieves leading planning performance among vision-only methods while maintaining high-fidelity action-controlled video generation capabilities, providing strong evidence for the effectiveness of unifying vision and motion representation for robust autonomous driving.

Related papers

• LVDrive: Latent Visual Representation Enhanced Vision-Language-Action Autonomous Driving Model [16.87741001074065]
  Vision-Language-Action (VLA) models have emerged as a promising framework for end-to-end autonomous driving.<n>Recent attempts to incorporate dense visual supervision via world modeling often overemphasize pixel-level image reconstruction.<n>We propose LVDrive, a Latent Visual representation enhanced VLA framework for autonomous driving.
  arXiv  Detail & Related papers  (2026-05-21T07:31:49Z)
• Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving [52.04950569530877]
  We present Uni-World VLA, a unified vision-language-action model that tightly interleaves future frame prediction and trajectory planning.<n>Our approach achieves competitive closed-loop planning performance while producing high-fidelity future frame predictions.
  arXiv  Detail & Related papers  (2026-03-28T14:39:51Z)
• DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving [49.11389494068169]
  We present DrivingGen, the first comprehensive benchmark for generative driving world models.<n>DrivingGen combines a diverse evaluation dataset curated from both driving datasets and internet-scale video sources.<n>General models look better but break physics, while driving-specific ones capture motion realistically but lag in visual quality.
  arXiv  Detail & Related papers  (2026-01-04T13:36:21Z)
• FutureX: Enhance End-to-End Autonomous Driving via Latent Chain-of-Thought World Model [103.2513470454204]
  FutureX is a pipeline that enhances end-to-end planners to perform complex motion planning via future scene latent reasoning and trajectory refinement.<n>FutureX enhances existing methods by producing more rational motion plans and fewer collisions without compromising efficiency.
  arXiv  Detail & Related papers  (2025-12-12T02:12:49Z)
• Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution [96.25314747309811]
  We introduce SeerDrive, a novel end-to-end framework that jointly models future scene evolution and trajectory planning.<n>Our method first predicts future bird's-eye view (BEV) representations to anticipate the dynamics of the surrounding scene.<n>Two key components enable this: (1) future-aware planning, which injects predicted BEV features into the trajectory planner, and (2) iterative scene modeling and vehicle planning.
  arXiv  Detail & Related papers  (2025-10-13T07:41:47Z)
• DriveX: Omni Scene Modeling for Learning Generalizable World Knowledge in Autonomous Driving [20.197094443215963]
  We present DriveX, a self-supervised world model that learns general scene dynamics and holistic representations from driving videos.<n>DriveX introduces Omni Scene Modeling (OSM), a module that unifies multimodal supervision-3D point cloud forecasting, 2D semantic representation, and image generation.<n>For downstream adaptation, we design Future Spatial Attention (FSA), a unified paradigm that dynamically aggregates features from DriveX's predictions to enhance task-specific inference.
  arXiv  Detail & Related papers  (2025-05-25T17:27:59Z)
• DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers [61.92571851411509]
  We introduce a multimodal driving language based on interleaved image and action tokens, and develop DrivingGPT to learn joint world modeling and planning.<n>Our DrivingGPT demonstrates strong performance in both action-conditioned video generation and end-to-end planning, outperforming strong baselines on large-scale nuPlan and NAVSIM benchmarks.
  arXiv  Detail & Related papers  (2024-12-24T18:59:37Z)
• Driving in the Occupancy World: Vision-Centric 4D Occupancy Forecasting and Planning via World Models for Autonomous Driving [15.100104512786107]
  Drive-OccWorld adapts a visioncentric- 4D forecasting world model to end-to-end planning for autonomous driving.<n>We propose injecting flexible action conditions, such as velocity, steering angle, trajectory, and commands, into the world model to enable controllable generation.<n>Our method can generate plausible and controllable 4D occupancy, paving the way for advancements in driving world generation and end-to-end planning.
  arXiv  Detail & Related papers  (2024-08-26T11:53:09Z)
• Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving [56.381918362410175]
  Drive-WM is the first driving world model compatible with existing end-to-end planning models. Our model generates high-fidelity multiview videos in driving scenes.
  arXiv  Detail & Related papers  (2023-11-29T18:59:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.