WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving

• URL: http://arxiv.org/abs/2608.20974v1
• Date: Fri, 21 Aug 2026 10:54:35 GMT
• Title: WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving
• Abstract summary: We present WA-JEPA, a VEPA-native world-action model for autonomous driving planning.<n>WA-JEPA employs hybrid future-masked pre-training, where the model infers future latents from observed context.<n>Results: WA-JEPA reaches 91.7S on NAVSIM, surpassing the strongest end-to-end and world-action baselines.
• Score: 8.510409798128272
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video Joint Embedding Predictive Architecture (V-JEPA) learns powerful spatiotemporal representations from video through self-supervised latent feature prediction. However, V-JEPA is built around random-mask completion and deterministic regression, making it fundamentally ill-suited for autonomous driving planning that demands future-directed prediction tightly coupled with action. To address this, we rethink the V-JEPA paradigm and present WA-JEPA, a V-JEPA-native world-action model designed for autonomous driving planning. Instead of random spatiotemporal masking, WA-JEPA employs hybrid future-masked pre-training, where the model infers future latents from observed context. Departing from deterministic regression, we recast future prediction as conditional flow matching over latent futures, which substantially improves the model's ability to generate plausible future latents for downstream planning. Finally, a joint future-action predictor is proposed to denoise future scene tokens and ego trajectories together in a unified spatiotemporal latent space, allowing action supervision to directly shape planning-relevant world representations. Pre-trained on nuPlan videos and fine-tuned on NAVSIM, WA-JEPA reaches 91.7 EPDMS on NAVSIM-v2, surpassing the strongest end-to-end and world-action baselines by 1.6 and 1.3 EPDMS, and, without HUGSIM-specific fine-tuning, attains the best HD-Score of 0.4462 on the closed-loop HUGSIM benchmark under the same evaluation protocol. These results validate V-JEPA-native world-action modeling as a powerful and scalable paradigm for autonomous driving planning. Code is available at https://github.com/AFARI-Research/WA-JEPA.

Related papers

• Flow-JEPA: Flow Matching for Robust Latent Dynamics in JEPA World Models [18.035166689862205]
  Joint-Embedding Predictive Architectures (JEPAs) have shown strong potential for learning compact predictive representations.<n>We propose Flow-JEPA, a conditional flow matching dynamics model that jointly generates a sequence of future latent states conditioned on the current observation and actions.
  arXiv  Detail & Related papers  (2026-08-29T03:40:06Z)
• Is Forward Prediction Enough? Physical State Grounding for JEPA World Models [18.174985055477666]
  PSG-JEPA shapes its latent space with two complementary grounding objectives beyond forward prediction.<n>We conduct experiments at three levels: (1) latent identifiability via probing, (2) goal-conditioned planning on frozen latents, and (3) policy learning in simulation and on a real robot.
  arXiv  Detail & Related papers  (2026-08-07T04:44:16Z)
• Auto-JEPA: A Latent World Model of Continuous Intent for End-to-End Autonomous Driving [3.174313305594326]
  We propose Auto-JEPA, an action-oriented latent world model that learns continuous future driving intent through joint-embedding prediction.<n>By optimizing only task-specific modules for trajectory representation, intent prediction, and candidate selection, Auto-JEPA achieves 91.3 PDMS on NAVSIM v1 and 89.1S on NAVSIM v2.
  arXiv  Detail & Related papers  (2026-07-31T05:10:23Z)
• AdaJEPA: An Adaptive Latent World Model [42.0759614364807]
  We propose AdaJEPA, an adaptive latent world model that performs test-time adaptation within the closed loop of model predictive control (MPC)<n>After training, AdaJEPA plans and executes the first action chunk, uses the observed next-state transition as a self-supervised adaptation signal, and replans with the updated model.<n>This closed-loop update continuously recalibrates the world model without additional expert demonstrations.
  arXiv  Detail & Related papers  (2026-06-30T17:53:48Z)
• IDOL: Inverse-Dynamics-Guided Future Prediction for End-to-End Autonomous Driving [6.515607168928518]
  IDOL is an inverse-dynamics-guided future prediction framework for world-model-based end-to-end planning in latent BEV space.<n>IDOL first predicts multiple future latent scene states with a BEV world model, then applies an inverse dynamics model to adjacent latent futures.<n>These inverse-dynamics-derived signals are used to optimize the planned trajectory, turning future forecasting from passive scene anticipation into actionable planning guidance.
  arXiv  Detail & Related papers  (2026-05-29T16:05:42Z)
• Uni-World VLA: Interleaved World Modeling and Planning for Autonomous Driving [52.04950569530877]
  We present Uni-World VLA, a unified vision-language-action model that tightly interleaves future frame prediction and trajectory planning.<n>Our approach achieves competitive closed-loop planning performance while producing high-fidelity future frame predictions.
  arXiv  Detail & Related papers  (2026-03-28T14:39:51Z)
• VGGT-World: Transforming VGGT into an Autoregressive Geometry World Model [59.789011777899965]
  VGGT-World is a geometry world model that side-steps video generation entirely and instead forecasts the temporal evolution of frozen geometry-foundation-model features.<n>We show that VGGT-World significantly outperforms the strongest baselines in depth forecasting while running 3.6-5 times faster with only 0.43B trainable parameters.
  arXiv  Detail & Related papers  (2026-03-13T04:56:43Z)
• Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving [26.905928731309572]
  Drive-JEPA is a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving.<n>First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning.<n>Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior.
  arXiv  Detail & Related papers  (2026-01-29T17:39:20Z)
• FutureX: Enhance End-to-End Autonomous Driving via Latent Chain-of-Thought World Model [103.2513470454204]
  FutureX is a pipeline that enhances end-to-end planners to perform complex motion planning via future scene latent reasoning and trajectory refinement.<n>FutureX enhances existing methods by producing more rational motion plans and fewer collisions without compromising efficiency.
  arXiv  Detail & Related papers  (2025-12-12T02:12:49Z)
• Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution [96.25314747309811]
  We introduce SeerDrive, a novel end-to-end framework that jointly models future scene evolution and trajectory planning.<n>Our method first predicts future bird's-eye view (BEV) representations to anticipate the dynamics of the surrounding scene.<n>Two key components enable this: (1) future-aware planning, which injects predicted BEV features into the trajectory planner, and (2) iterative scene modeling and vehicle planning.
  arXiv  Detail & Related papers  (2025-10-13T07:41:47Z)
• End-to-End Driving with Online Trajectory Evaluation via BEV World Model [52.10633338584164]
  We propose an end-to-end driving framework WoTE, which leverages a BEV World model to predict future BEV states for Trajectory Evaluation.<n>We validate our framework on the NAVSIM benchmark and the closed-loop Bench2Drive benchmark based on the CARLA simulator, achieving state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-04-02T17:47:23Z)
• Driving into the Future: Multiview Visual Forecasting and Planning with World Model for Autonomous Driving [56.381918362410175]
  Drive-WM is the first driving world model compatible with existing end-to-end planning models. Our model generates high-fidelity multiview videos in driving scenes.
  arXiv  Detail & Related papers  (2023-11-29T18:59:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.