Video2Track: From Real-World Interaction Videos to Steerable Adversarial Closed-Track Testing for Automated Driving Systems

• URL: http://arxiv.org/abs/2608.11592v1
• Date: Wed, 12 Aug 2026 03:07:00 GMT
• Title: Video2Track: From Real-World Interaction Videos to Steerable Adversarial Closed-Track Testing for Automated Driving Systems
• Abstract summary: Video2Track is a framework that transfers real-world interactive driving scenarios from videos into steerable adversarial closed-track testing.<n>The proposed framework can faithfully reproduce representative real-world interaction scenarios and generate executable scenario variants with controllable risk levels and interaction styles.
• Score: 11.663429828510786
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Closed-track testing plays a fundamental role in the verification and validation of automated driving systems (ADS), particularly for safety-critical scenarios, by enabling reproducible evaluation under controlled conditions. However, most existing approaches still rely on standardized protocols or predefined trajectories, leading to overly scripted interactions and limited ability to reproduce the natural complexity of public-road traffic. To address this limitation, we propose Video2Track, a framework that transfers real-world interactive driving scenarios from videos into steerable adversarial closed-track testing. The framework consists of two tightly coupled modules. The first is a scenario semantic mapping module, which extracts structured semantics from driving videos using a vision-language model and grounds them onto a closed-track topology library via retrieval-augmented generation, thereby identifying compatible map segments and interaction anchors. The second is a dynamic interactive testing module, which conditions on the grounded topology and anchors to generate diverse multi-agent trajectories through a conditional diffusion model, while regulating interaction intensity via a Stackelberg game with a parameterized adversarial objective. Closed-track experiments demonstrate that the proposed framework can faithfully reproduce representative real-world interaction scenarios and generate executable scenario variants with controllable risk levels and interaction styles, providing a scalable approach for realistic and steerable ADS validation.

Related papers

• End-to-end Conditional Diffusion for Realistic and Controllable Visual Traffic Scenario Generation [59.188006170485046]
  E2E-CDiff is an end-to-end conditional diffusion framework for controllable and realistic scenario generation.<n>Differentiable guidance regulates speed, enforces drivable-area compliance, and supports collision-avoidance or collision-seeking behaviors.<n>E2E-CDiff also performs competitively as a learning-based ego planner.
  arXiv  Detail & Related papers  (2026-07-21T02:13:25Z)
• V2XCrafter: Learning to Generate Driving Scene Across Agents [52.22336670088219]
  Collaborative driving systems leverage vehicle-to-everything (V2X) communication for collaborative perception to enhance driving safety.<n>We propose V2XCrafter, the first framework for generating controllable and realistic collaborative driving scene across agents' camera views.<n>Experiments have shown that V2XCrafter can generate high-fidelity and controllable street views with consistency across agents, thereby effectively enhancing the downstream collaborative 3D object detection tasks.
  arXiv  Detail & Related papers  (2026-05-28T07:03:51Z)
• MAPLE: Latent Multi-Agent Play for End-to-End Autonomous Driving [62.43744546817599]
  Vision-language--action (VLA) models are effective as end-to-end motion planners, but can be brittle when evaluated in closed-loop settings.<n>We propose MAPLE, a novel framework for reactive, multi-agent rollout of a dynamic driving scenario in the latent space of the VLA model.<n> MAPLE achieves state-of-the-art driving performance on Bench2Drive and demonstrates scalable, closed-loop multi-agent play for robust E2E autonomous driving systems.
  arXiv  Detail & Related papers  (2026-05-13T23:35:14Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• An LLM-driven Scenario Generation Pipeline Using an Extended Scenic DSL for Autonomous Driving Safety Validation [4.602386383455713]
  Real-world crash reports are valuable for scenario-based testing of autonomous driving systems.<n>Current methods cannot effectively translate this multimodal data into precise, executable simulation scenarios.<n>We propose a scalable and verifiable pipeline that uses a large language model and a probabilistic intermediate representation.
  arXiv  Detail & Related papers  (2026-02-24T07:44:26Z)
• Referring Video Object Segmentation with Cross-Modality Proxy Queries [23.504655272754587]
  Referring video object segmentation (RVOS) is an emerging cross-modality task that aims to generate pixel-level maps of the target objects referred by given textual expressions.<n>Recent approaches address cross-modality alignment through conditional queries, tracking the target object using a query-response based mechanism.<n>We propose a novel RVOS architecture called ProxyFormer, which introduces a set of proxy queries to integrate visual and text semantics.
  arXiv  Detail & Related papers  (2025-11-26T07:45:41Z)
• LLM-based Realistic Safety-Critical Driving Video Generation [4.537331974356809]
  We propose a framework that automatically synthesizes driving scenarios within the CARLA simulator.<n>The framework has flexibility in scenario scripting, efficient code-based control of traffic participants, and enforcement of realistic physical dynamics.<n>Our approach enables controllable scenario generation and facilitates the creation of rare but critical edge cases.
  arXiv  Detail & Related papers  (2025-07-02T00:45:19Z)
• LANGTRAJ: Diffusion Model and Dataset for Language-Conditioned Trajectory Simulation [102.1527101235251]
  LangTraj is a language-conditioned scene-diffusion model that simulates the joint behavior of all agents in traffic scenarios.<n>By conditioning on natural language inputs, LangTraj provides flexible and intuitive control over interactive behaviors.<n>LangTraj demonstrates strong performance in realism, language controllability, and language-conditioned safety-critical simulation.
  arXiv  Detail & Related papers  (2025-04-15T17:14:06Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• TARGET: Automated Scenario Generation from Traffic Rules for Testing Autonomous Vehicles via Validated LLM-Guided Knowledge Extraction [8.029974249105443]
  TARGET is an end-to-end framework that automatically generates test scenarios from traffic rules.<n>We leverage a Large Language Model (LLM) to extract knowledge from traffic rules.<n>TARGET synthesizes executable scripts to render scenarios in simulation.
  arXiv  Detail & Related papers  (2023-05-10T10:04:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.