Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges

• URL: http://arxiv.org/abs/2603.09086v1
• Date: Tue, 10 Mar 2026 01:56:17 GMT
• Title: Latent World Models for Automated Driving: A Unified Taxonomy, Evaluation Framework, and Open Challenges
• Abstract summary: This paper proposes a unifying latent-space framework that synthesizes recent progress in world models for automated driving.<n>The framework organizes the design space by the target and form of latent representations (latent worlds, latent actions, latent generators; continuous states, discrete tokens, and hybrids) and by structural priors for geometry, topology, and semantics.
• Score: 2.76240219662896
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Emerging generative world models and vision-language-action (VLA) systems are rapidly reshaping automated driving by enabling scalable simulation, long-horizon forecasting, and capability-rich decision making. Across these directions, latent representations serve as the central computational substrate: they compress high-dimensional multi-sensor observations, enable temporally coherent rollouts, and provide interfaces for planning, reasoning, and controllable generation. This paper proposes a unifying latent-space framework that synthesizes recent progress in world models for automated driving. The framework organizes the design space by the target and form of latent representations (latent worlds, latent actions, latent generators; continuous states, discrete tokens, and hybrids) and by structural priors for geometry, topology, and semantics. Building on this taxonomy, the paper articulates five cross-cutting internal mechanics (i.e, structural isomorphism, long-horizon temporal stability, semantic and reasoning alignment, value-aligned objectives and post-training, as well as adaptive computation and deliberation) and connects these design choices to robustness, generalization, and deployability. The work also proposes concrete evaluation prescriptions, including a closed-loop metric suite and a resource-aware deliberation cost, designed to reduce the open-loop / closed-loop mismatch. Finally, the paper identifies actionable research directions toward advancing latent world model for decision-ready, verifiable, and resource-efficient automated driving.

Related papers

• Planning-Oriented End-to-End Autonomous Driving: Architectures, Evaluation, and Emerging Paradigms [33.863182759843845]
  End-to-end autonomous driving has evolved from camera-to-control regression toward planning-oriented systems.<n>This survey reviews this transition across behavior cloning, conditional imitation learning, privileged distillation, BEV and vectorized planning.<n>We argue that the key distinction in modern end-to-end driving is not whether intermediate representations are used, but whether they are learned, supervised, and evaluated to support safe, feasible, and route-compliant planning.
  arXiv  Detail & Related papers  (2026-08-20T14:41:36Z)
• Orthogonal JEPA: Factorized Predictive States for Latent World Models [95.20158869896393]
  We introduce method, a latent world-modeling framework based on predictive factorization.<n>We show that method can allocate redundant capacity to dominant signals while providing weak or conflicting gradients to less dominant predictive structure.<n>Experiments on controlled vision, single-cell transcriptomics, longitudinal health records, continuous control, and molecular dynamics evaluate representation quality, forecasting, planning, and long-horizon stability.
  arXiv  Detail & Related papers  (2026-08-20T13:59:57Z)
• CascadeOcc: Rethinking 3D Occupancy World Models with Cascaded VQ Representations [27.127669455054754]
  CascadeOcc is a novel occupancy world model for autonomous driving.<n>It prioritizes intrinsic structural hierarchy over extrinsic auxiliary modalities.<n>We show that CascadeOcc achieves superior performance among vision-centric approaches.
  arXiv  Detail & Related papers  (2026-06-26T01:43:37Z)
• Lagrange: An Open-Vocabulary, Energy-Based Sparse Framework for Generalized End-to-End Driving [12.828441606176447]
  Scaling end-to-end autonomous driving to complex, open-world environments requires perceptual models that generalize to anomalous scenarios and planners that produce kinematically valid trajectories.<n>We propose Lagrange, an open-vocabulary, computationally sparse driving framework based on Masked Latent Fields (MLF)<n>We introduce an intent-driven masked cross-attention module that temporally filters irrelevant entities, decoding the attended tokens into an implicit continuous energy field defined over spatial coordinates.
  arXiv  Detail & Related papers  (2026-06-18T14:18:01Z)
• GraphWorld: Long-Horizon Planning with World Models for End-to-End Autonomous Driving [56.50032398571235]
  GraphWorld is an E2E-AD framework that explicitly enhances long-horizon planning through latent world modeling.<n>We present a World-State-Conditioned Planning that learns ego-centric latent world representations by modeling interactions between an ego vehicle and surrounding agents.<n>This latent world state captures key interaction dynamics and safety-relevant semantics, and serves as a conditioning signal to guide long-horizon, safety-aware trajectory planning.
  arXiv  Detail & Related papers  (2026-06-15T06:26:46Z)
• ManipArena: Comprehensive Real-world Evaluation of Reasoning-Oriented Generalist Robot Manipulation [66.02142169323521]
  Vision-Language-Action models and world models have recently emerged as promising paradigms for general-purpose robotic intelligence.<n>Existing benchmarks are largely simulator-centric, which provide controllability but fail to capture the reality gap caused by perception noise.<n>We introduce ManipArena, a standardized evaluation framework designed to bridge simulation and real-world execution.
  arXiv  Detail & Related papers  (2026-03-30T15:06:41Z)
• Kinematics-Aware Latent World Models for Data-Efficient Autonomous Driving [8.391716372682316]
  Vehicle kinematic information is incorporated into the observation encoder to ground latent transitions in physically meaningful motion dynamics.<n> geometry-aware supervision regularizes the RSSM latent state to capture task-relevant spatial structure beyond pixel reconstruction.
  arXiv  Detail & Related papers  (2026-03-07T15:47:54Z)
• Specification-Driven Generation and Evaluation of Discrete-Event World Models via the DEVS Formalism [9.578714623643291]
  We seek a principled middle ground that combines the reliability of explicit simulators with the flexibility of learned models.<n>We advocate explicit, executable discrete-event world models synthesized directly from natural-language specifications.
  arXiv  Detail & Related papers  (2026-03-04T06:50:32Z)
• Forging Spatial Intelligence: A Roadmap of Multi-Modal Data Pre-Training for Autonomous Systems [75.78934957242403]
  Self-driving vehicles and drones require true Spatial Intelligence from multi-modal onboard sensor data.<n>This paper presents a framework for multi-modal pre-training, identifying the core set of techniques driving progress toward this goal.
  arXiv  Detail & Related papers  (2025-12-30T17:58:01Z)
• A Comprehensive Survey on World Models for Embodied AI [14.457261562275121]
  Embodied AI requires agents that perceive, act, and anticipate how actions reshape future world states.<n>This survey presents a unified framework for world models in embodied AI.
  arXiv  Detail & Related papers  (2025-10-19T07:12:32Z)
• ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving [49.07731497951963]
  ReCogDrive is a novel Reinforced Cognitive framework for end-to-end autonomous driving.<n>We introduce a hierarchical data pipeline that mimics the sequential cognitive process of human drivers.<n>We then address the language-action mismatch by injecting the VLM's learned driving priors into a diffusion planner.
  arXiv  Detail & Related papers  (2025-06-09T03:14:04Z)
• A Survey of World Models for Autonomous Driving [55.520179689933904]
  Recent breakthroughs in autonomous driving have been propelled by advances in robust world modeling.<n>World models offer high-fidelity representations of the driving environment that integrate multi-sensor data, semantic cues, and temporal dynamics.<n>Future research must address key challenges in self-supervised representation learning, multimodal fusion, and advanced simulation.
  arXiv  Detail & Related papers  (2025-01-20T04:00:02Z)
• Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey [61.39993881402787]
  World models and video generation are pivotal technologies in the domain of autonomous driving. This paper investigates the relationship between these two technologies. By analyzing the interplay between video generation and world models, this survey identifies critical challenges and future research directions.
  arXiv  Detail & Related papers  (2024-11-05T08:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.