Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving

• URL: http://arxiv.org/abs/2601.22032v1
• Date: Thu, 29 Jan 2026 17:39:20 GMT
• Title: Drive-JEPA: Video JEPA Meets Multimodal Trajectory Distillation for End-to-End Driving
• Abstract summary: Drive-JEPA is a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving.<n>First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning.<n>Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior.
• Score: 26.905928731309572
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: End-to-end autonomous driving increasingly leverages self-supervised video pretraining to learn transferable planning representations. However, pretraining video world models for scene understanding has so far brought only limited improvements. This limitation is compounded by the inherent ambiguity of driving: each scene typically provides only a single human trajectory, making it difficult to learn multimodal behaviors. In this work, we propose Drive-JEPA, a framework that integrates Video Joint-Embedding Predictive Architecture (V-JEPA) with multimodal trajectory distillation for end-to-end driving. First, we adapt V-JEPA for end-to-end driving, pretraining a ViT encoder on large-scale driving videos to produce predictive representations aligned with trajectory planning. Second, we introduce a proposal-centric planner that distills diverse simulator-generated trajectories alongside human trajectories, with a momentum-aware selection mechanism to promote stable and safe behavior. When evaluated on NAVSIM, the V-JEPA representation combined with a simple transformer-based decoder outperforms prior methods by 3 PDMS in the perception-free setting. The complete Drive-JEPA framework achieves 93.3 PDMS on v1 and 87.8 EPDMS on v2, setting a new state-of-the-art.

Related papers

• WA-JEPA: Rethinking the Video JEPA Paradigm for World-Action Modeling in Autonomous Driving [8.510409798128272]
  We present WA-JEPA, a VEPA-native world-action model for autonomous driving planning.<n>WA-JEPA employs hybrid future-masked pre-training, where the model infers future latents from observed context.<n>Results: WA-JEPA reaches 91.7S on NAVSIM, surpassing the strongest end-to-end and world-action baselines.
  arXiv  Detail & Related papers  (2026-08-21T10:54:35Z)
• SimWAM: A Simple World Action Model for End-to-End Autonomous Driving [66.93548079427477]
  We present SimWAM, a simple yet effective WAM that leverages future-video prediction as a training-time supervision signal.<n>An isolated attention mask keeps action prediction independent of future frames, allowing trajectory prediction without explicit future-frame generation at inference.<n>Our SimWAM achieves 91.5 PDMS on NAVSIM, surpasses state-of-the-art WAM-based planners with substantially lower latency, and transfers zero-shot to nuScenes.
  arXiv  Detail & Related papers  (2026-08-07T17:59:09Z)
• SUV: Future Scene Understanding as Video Generation for End-to-End Driving [71.39332228514006]
  We introduce SUV, a unified end-to-end driving framework that casts future Scene Understanding as Video generation.<n>SUV models future appearance, semantics, relative depth, and instance-level dynamics as video streams with a shared video expert.<n>Experiments show that SUV directly predicts all four future streams, while controlled ablations show that structured future supervision and direct future-stream access yield higher trajectory planning scores.
  arXiv  Detail & Related papers  (2026-08-04T03:51:53Z)
• DriveWAM: Video Generative Priors Enable Scalable World-Action Modeling for Autonomous Driving [19.231942908215174]
  We present DriveWAM, a driving world-action model that adapts a pretrained video diffusion transformer into an autoregressive video-action policy.<n>DriveWAM organizes video and action streams into a unified temporal token sequence and trains them under a joint flow-matching objective.<n>Experiments on NAVSIM and the PhysicalAI-Autonomous-Vehicles benchmark show that DriveWAM achieves strong planning performance.
  arXiv  Detail & Related papers  (2026-05-27T14:36:26Z)
• LMGenDrive: Bridging Multimodal Understanding and Generative World Modeling for End-to-End Driving [60.31765454895336]
  We present LMGenDrive, the first framework that combines multimodal understanding with generative world models for end-to-end closed-loop driving.<n>We propose a progressive three-stage training strategy, from vision pretraining to multi-step long-horizon driving, to improve stability and performance.
  arXiv  Detail & Related papers  (2026-04-09T19:13:14Z)
• DriveVA: Video Action Models are Zero-Shot Drivers [23.675212404889805]
  Generalization is a central challenge in autonomous driving, as real-world scenarios require robust performance under unseen conditions.<n>Recent world-model-based planning methods have shown strong capabilities in scene understanding and multi-modal future prediction.<n>We propose DriveVA, a novel autonomous driving world model that jointly decodes future visual forecasts and action sequences in a shared latent generative process.
  arXiv  Detail & Related papers  (2026-04-05T17:43:16Z)
• DriveLaW:Unifying Planning and Video Generation in a Latent Driving World [40.4279880486975]
  We propose DriveLaW, a novel paradigm that unifies video generation and motion planning.<n>DriveLaW consists of two core components: DriveLaW-Video, our powerful world model that generates high-fidelity forecasting with expressive latent representations, and DriveLaW-Act, a diffusion planner.<n>DriveLaW not only advances video prediction significantly, surpassing best-performing work by 33.3% in FID and 1.8% in FVD, but also achieves a new record on the NAVSIM planning benchmark.
  arXiv  Detail & Related papers  (2025-12-29T12:32:27Z)
• GenieDrive: Towards Physics-Aware Driving World Model with 4D Occupancy Guided Video Generation [80.1493315900789]
  We propose GenieDrive, a framework for physics-aware driving video generation.<n>Our approach starts by generating 4D occupancy, which serves as a physics-informed foundation for subsequent video generation.<n>Experiments demonstrate that GenieDrive enables highly controllable, multi-view consistent, and physics-aware driving video generation.
  arXiv  Detail & Related papers  (2025-12-14T16:23:51Z)
• DriveVGGT: Visual Geometry Transformer for Autonomous Driving [50.5036123750788]
  DriveVGGT is a scale-aware 4D reconstruction framework specifically designed for autonomous driving data.<n>We propose a temporal Video Attention (TVA) module to process multi-camera videos independently.<n>Then, we propose a Multi-camera Consistency Attention (MCA) module to conduct window attention with normalized relative pose embeddings.
  arXiv  Detail & Related papers  (2025-11-27T09:40:43Z)
• VR-Drive: Viewpoint-Robust End-to-End Driving with Feed-Forward 3D Gaussian Splatting [47.78433964322689]
  VR-Drive is a novel E2E-AD framework that addresses viewpoint generalization by jointly learning 3D scene reconstruction.<n>Our results demonstrate that VR-Drive is a scalable and robust solution for the real-world deployment of end-to-end autonomous driving systems.
  arXiv  Detail & Related papers  (2025-10-27T10:49:39Z)
• Future-Aware End-to-End Driving: Bidirectional Modeling of Trajectory Planning and Scene Evolution [96.25314747309811]
  We introduce SeerDrive, a novel end-to-end framework that jointly models future scene evolution and trajectory planning.<n>Our method first predicts future bird's-eye view (BEV) representations to anticipate the dynamics of the surrounding scene.<n>Two key components enable this: (1) future-aware planning, which injects predicted BEV features into the trajectory planner, and (2) iterative scene modeling and vehicle planning.
  arXiv  Detail & Related papers  (2025-10-13T07:41:47Z)
• Delta-Triplane Transformers as Occupancy World Models [57.16979927973973]
  Occupancy World Models (OWMs) aim to predict future scenes via 3D voxelized representations of the environment to support intelligent motion planning.<n>We propose Delta-Triplane Transformers (DTT), a novel 4D OWM for autonomous driving, that introduces two key innovations.<n>DTT delivers a 1.44$times$ speedup (26 FPS) over the state of the art, improves mean IoU to 30.85, and reduces the mean absolute planning error to 1.0 meters.
  arXiv  Detail & Related papers  (2025-03-10T13:50:23Z)
• DiFSD: Ego-Centric Fully Sparse Paradigm with Uncertainty Denoising and Iterative Refinement for Efficient End-to-End Self-Driving [55.53171248839489]
  We propose an ego-centric fully sparse paradigm, named DiFSD, for end-to-end self-driving.<n>Specifically, DiFSD mainly consists of sparse perception, hierarchical interaction and iterative motion planner.<n>Experiments conducted on nuScenes and Bench2Drive datasets demonstrate the superior planning performance and great efficiency of DiFSD.
  arXiv  Detail & Related papers  (2024-09-15T15:55:24Z)
• DriveScape: Towards High-Resolution Controllable Multi-View Driving Video Generation [10.296670127024045]
  DriveScape is an end-to-end framework for multi-view, 3D condition-guided video generation. Our Bi-Directional Modulated Transformer (BiMot) ensures precise alignment of 3D structural information. DriveScape excels in video generation performance, achieving state-of-the-art results on the nuScenes dataset with an FID score of 8.34 and an FVD score of 76.39.
  arXiv  Detail & Related papers  (2024-09-09T09:43:17Z)
• ADAPT: Action-aware Driving Caption Transformer [24.3857045947027]
  We propose an end-to-end transformer-based architecture, ADAPT, which provides user-friendly natural language narrations and reasoning for each decision making step of autonomous vehicular control and action. Experiments on BDD-X dataset demonstrate state-of-the-art performance of the ADAPT framework on both automatic metrics and human evaluation. To illustrate the feasibility of the proposed framework in real-world applications, we build a novel deployable system that takes raw car videos as input and outputs the action narrations and reasoning in real time.
  arXiv  Detail & Related papers  (2023-02-01T18:59:19Z)
• Policy Pre-training for End-to-end Autonomous Driving via Self-supervised Geometric Modeling [96.31941517446859]
  We propose PPGeo (Policy Pre-training via Geometric modeling), an intuitive and straightforward fully self-supervised framework curated for the policy pretraining in visuomotor driving. We aim at learning policy representations as a powerful abstraction by modeling 3D geometric scenes on large-scale unlabeled and uncalibrated YouTube driving videos. In the first stage, the geometric modeling framework generates pose and depth predictions simultaneously, with two consecutive frames as input. In the second stage, the visual encoder learns driving policy representation by predicting the future ego-motion and optimizing with the photometric error based on current visual observation only.
  arXiv  Detail & Related papers  (2023-01-03T08:52:49Z)
• MultiXNet: Multiclass Multistage Multimodal Motion Prediction [27.046311751308775]
  MultiXNet is an end-to-end approach for detection and motion prediction based directly on lidar sensor data. The method was evaluated on large-scale, real-world data collected by a fleet of SDVs in several cities.
  arXiv  Detail & Related papers  (2020-06-03T01:01:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.