Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models

• URL: http://arxiv.org/abs/2510.06209v1
• Date: Tue, 07 Oct 2025 17:58:32 GMT
• Title: Drive&Gen: Co-Evaluating End-to-End Driving and Video Generation Models
• Authors: Jiahao Wang, Zhenpei Yang, Yijing Bai, Yingwei Li, Yuliang Zou, Bo Sun, Abhijit Kundu, Jose Lezama, Luna Yue Huang, Zehao Zhu, Jyh-Jing Hwang, Dragomir Anguelov, Mingxing Tan, Chiyu Max Jiang,
• Abstract summary: We propose novel statistical measures leveraging E2E drivers to evaluate the realism of generated videos.<n>We show that synthetic data produced by the video generation model offers a cost-effective alternative to real-world data collection.
• Score: 33.32483442886097
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in generative models have sparked exciting new possibilities in the field of autonomous vehicles. Specifically, video generation models are now being explored as controllable virtual testing environments. Simultaneously, end-to-end (E2E) driving models have emerged as a streamlined alternative to conventional modular autonomous driving systems, gaining popularity for their simplicity and scalability. However, the application of these techniques to simulation and planning raises important questions. First, while video generation models can generate increasingly realistic videos, can these videos faithfully adhere to the specified conditions and be realistic enough for E2E autonomous planner evaluation? Second, given that data is crucial for understanding and controlling E2E planners, how can we gain deeper insights into their biases and improve their ability to generalize to out-of-distribution scenarios? In this work, we bridge the gap between the driving models and generative world models (Drive&Gen) to address these questions. We propose novel statistical measures leveraging E2E drivers to evaluate the realism of generated videos. By exploiting the controllability of the video generation model, we conduct targeted experiments to investigate distribution gaps affecting E2E planner performance. Finally, we show that synthetic data produced by the video generation model offers a cost-effective alternative to real-world data collection. This synthetic data effectively improves E2E model generalization beyond existing Operational Design Domains, facilitating the expansion of autonomous vehicle services into new operational contexts.

Related papers

• Causal World Modeling for Robot Control [56.31803788587547]
  Video world models provide the ability to imagine the near future by understanding the causality between actions and visual dynamics.<n>We introduce LingBot-VA, an autoregressive diffusion framework that learns frame prediction and policy execution simultaneously.<n>We evaluate our model on both simulation benchmarks and real-world scenarios, where it shows significant promise in long-horizon manipulation, data efficiency in post-training, and strong generalizability to novel configurations.
  arXiv  Detail & Related papers  (2026-01-29T17:07:43Z)
• DrivingGen: A Comprehensive Benchmark for Generative Video World Models in Autonomous Driving [49.11389494068169]
  We present DrivingGen, the first comprehensive benchmark for generative driving world models.<n>DrivingGen combines a diverse evaluation dataset curated from both driving datasets and internet-scale video sources.<n>General models look better but break physics, while driving-specific ones capture motion realistically but lag in visual quality.
  arXiv  Detail & Related papers  (2026-01-04T13:36:21Z)
• SynAD: Enhancing Real-World End-to-End Autonomous Driving Models through Synthetic Data Integration [18.10769055616004]
  We introduce SynAD, the first framework designed to enhance real-world E2E AD models using synthetic data.<n>Our method designates the agent with the most comprehensive driving information as the ego vehicle in a multi-agent synthetic scenario.<n>We devise a training strategy that effectively integrates these map-based synthetic data with real driving data.
  arXiv  Detail & Related papers  (2025-10-28T04:22:02Z)
• A Survey of World Models for Autonomous Driving [55.520179689933904]
  Recent breakthroughs in autonomous driving have been propelled by advances in robust world modeling.<n>World models offer high-fidelity representations of the driving environment that integrate multi-sensor data, semantic cues, and temporal dynamics.<n>Future research must address key challenges in self-supervised representation learning, multimodal fusion, and advanced simulation.
  arXiv  Detail & Related papers  (2025-01-20T04:00:02Z)
• DrivingGPT: Unifying Driving World Modeling and Planning with Multi-modal Autoregressive Transformers [61.92571851411509]
  We introduce a multimodal driving language based on interleaved image and action tokens, and develop DrivingGPT to learn joint world modeling and planning.<n>Our DrivingGPT demonstrates strong performance in both action-conditioned video generation and end-to-end planning, outperforming strong baselines on large-scale nuPlan and NAVSIM benchmarks.
  arXiv  Detail & Related papers  (2024-12-24T18:59:37Z)
• Exploring the Interplay Between Video Generation and World Models in Autonomous Driving: A Survey [61.39993881402787]
  World models and video generation are pivotal technologies in the domain of autonomous driving. This paper investigates the relationship between these two technologies. By analyzing the interplay between video generation and world models, this survey identifies critical challenges and future research directions.
  arXiv  Detail & Related papers  (2024-11-05T08:58:35Z)
• EVA: An Embodied World Model for Future Video Anticipation [30.721105710709008]
  Video generation models have made significant progress in simulating future states, showcasing their potential as world simulators in embodied scenarios.<n>Existing models often lack robust understanding, limiting their ability to perform multi-step predictions or handle Out-of-Distribution (OOD) scenarios.<n>We propose the Reflection of Generation (RoG), a set of intermediate reasoning strategies designed to enhance video prediction.
  arXiv  Detail & Related papers  (2024-10-20T18:24:00Z)
• DrivingDojo Dataset: Advancing Interactive and Knowledge-Enriched Driving World Model [65.43473733967038]
  We introduce DrivingDojo, the first dataset tailor-made for training interactive world models with complex driving dynamics. Our dataset features video clips with a complete set of driving maneuvers, diverse multi-agent interplay, and rich open-world driving knowledge.
  arXiv  Detail & Related papers  (2024-10-14T17:19:23Z)
• GenDDS: Generating Diverse Driving Video Scenarios with Prompt-to-Video Generative Model [6.144680854063938]
  GenDDS is a novel approach for generating driving scenarios for autonomous driving systems. We employ the KITTI dataset, which includes real-world driving videos, to train the model. We demonstrate that our model can generate high-quality driving videos that closely replicate the complexity and variability of real-world driving scenarios.
  arXiv  Detail & Related papers  (2024-08-28T15:37:44Z)
• GenAD: Generalized Predictive Model for Autonomous Driving [75.39517472462089]
  We introduce the first large-scale video prediction model in the autonomous driving discipline. Our model, dubbed GenAD, handles the challenging dynamics in driving scenes with novel temporal reasoning blocks. It can be adapted into an action-conditioned prediction model or a motion planner, holding great potential for real-world driving applications.
  arXiv  Detail & Related papers  (2024-03-14T17:58:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.