From Driving Videos to Simulatable Scenarios

• URL: http://arxiv.org/abs/2606.21993v1
• Date: Sat, 20 Jun 2026 11:16:28 GMT
• Title: From Driving Videos to Simulatable Scenarios
• Abstract summary: This paper introduces D-V2S, a framework that automatically generates simulatable driving scenarios from driving videos.<n>We show that D-V2S generates scenarios where 90% of the relevant semantic elements of the videos are present.<n>We also provide both semantic and human driven ablative analyses of D-V2S's modules.
• Score: 41.99844472131922
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Autonomous vehicles (AVs) face driving scenarios ranging from routine traffic to rare events. To assess safety it is crucial to reproduce these scenarios in a controllable, repeatable, and scalable manner, with simulation playing a key role. This paper introduces D-V2S, a novel framework that automatically generates simulatable driving scenarios from driving videos. D-V2S operates in two stages: a Driving Record Analyzer (DRA) uses a vision language model (VLM) with our designed prompt to produce natural-language descriptions from input videos, capturing road layouts and dynamic traffic interactions; subsequently, a Scenario Generator (SG) uses a large language model (LLM) and our conditioning context to translate these descriptions into executable scenarios. Using simulations, we show that D-V2S generates scenarios where 90% of the relevant semantic elements of the videos are present. We also provide qualitative results demonstrating D-V2S's capability to transform real-world driving videos into simulatable scenarios. Moreover, we provide both semantic and human driven ablative analyses of D-V2S's modules. In particular, we show how the VLM choice matters for DRA, and how our SG achieves a 75% preference rate over other state-of-the-art methods.

Related papers

• TRANSPORTER: Transferring Visual Semantics from VLM Manifolds [56.749972238005604]
  This paper introduces a logits-to-video (L2V) task alongside a model-independent approach, TRANSPORTER, to generate videos.<n> TRANSPORTER learns an optimal transport coupling to VLM's high-semantic embedding spaces.<n>In turn, logit scores define embedding directions for conditional video generation.
  arXiv  Detail & Related papers  (2025-11-23T09:12:48Z)
• LLM-based Realistic Safety-Critical Driving Video Generation [4.537331974356809]
  We propose a framework that automatically synthesizes driving scenarios within the CARLA simulator.<n>The framework has flexibility in scenario scripting, efficient code-based control of traffic participants, and enforcement of realistic physical dynamics.<n>Our approach enables controllable scenario generation and facilitates the creation of rare but critical edge cases.
  arXiv  Detail & Related papers  (2025-07-02T00:45:19Z)
• VaViM and VaVAM: Autonomous Driving through Video Generative Modeling [88.33638585518226]
  We introduce an open-source auto-regressive video model (VaM) and its companion video-action model (VaVAM) to investigate how video pre-training transfers to real-world driving.<n>We evaluate our models in open- and closed-loop driving scenarios, revealing that video-based pre-training holds promise for autonomous driving.
  arXiv  Detail & Related papers  (2025-02-21T18:56:02Z)
• Generating Out-Of-Distribution Scenarios Using Language Models [58.47597351184034]
  Large Language Models (LLMs) have shown promise in autonomous driving. This paper introduces a framework for generating diverse Out-Of-Distribution (OOD) driving scenarios. We evaluate our framework through extensive simulations and introduce a new "OOD-ness" metric.
  arXiv  Detail & Related papers  (2024-11-25T16:38:17Z)
• From Dashcam Videos to Driving Simulations: Stress Testing Automated Vehicles against Rare Events [5.132984904858975]
  Testing Automated Driving Systems (ADS) in simulation with realistic driving scenarios is important for verifying their performance.<n>We propose a novel framework that automates the conversion of real-world car crash videos into detailed simulation scenarios.<n>Our preliminary results demonstrate substantial time efficiency, finishing the real-to-sim conversion in minutes with full automation and no human intervention.
  arXiv  Detail & Related papers  (2024-11-25T01:01:54Z)
• SimpleLLM4AD: An End-to-End Vision-Language Model with Graph Visual Question Answering for Autonomous Driving [15.551625571158056]
  We propose an e2eAD method called SimpleLLM4AD. In our method, the e2eAD task are divided into four stages, which are perception, prediction, planning, and behavior. Our experiments demonstrate that SimpleLLM4AD achieves competitive performance in complex driving scenarios.
  arXiv  Detail & Related papers  (2024-07-31T02:35:33Z)
• ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles [17.396416459648755]
  ChatScene is a Large Language Model (LLM)-based agent that generates safety-critical scenarios for autonomous vehicles. A key part of our agent is a comprehensive knowledge retrieval component, which efficiently translates specific textual descriptions into corresponding domain-specific code snippets.
  arXiv  Detail & Related papers  (2024-05-22T23:21:15Z)
• Probing Multimodal LLMs as World Models for Driving [72.18727651074563]
  We look at the application of Multimodal Large Language Models (MLLMs) in autonomous driving. Despite advances in models like GPT-4o, their performance in complex driving environments remains largely unexplored.
  arXiv  Detail & Related papers  (2024-05-09T17:52:42Z)
• Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles [86.9067793493874]
  We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
  arXiv  Detail & Related papers  (2021-03-12T17:00:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.