ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation

• URL: http://arxiv.org/abs/2604.17147v1
• Date: Sat, 18 Apr 2026 21:00:26 GMT
• Title: ScenarioControl: Vision-Language Controllable Vectorized Latent Scenario Generation
• Abstract summary: Given a text prompt or an input image, Scenario-Control synthesizes diverse, realistic 3D scenario rollouts.<n>The method generates scenes in a vectorized latent space that represents road structure and dynamic agents jointly.
• Score: 33.56680018838822
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: We introduce ScenarioControl, the first vision-language control mechanism for learned driving scenario generation. Given a text prompt or an input image, Scenario-Control synthesizes diverse, realistic 3D scenario rollouts - including map, 3D boxes of reactive actors over time, pedestrians, driving infrastructure, and ego camera observations. The method generates scenes in a vectorized latent space that represents road structure and dynamic agents jointly. To connect multimodal control with sparse vectorized scene elements, we propose a cross-global control mechanism that integrates crossattention with a lightweight global-context branch, enabling fine-grained control over road layout and traffic conditions while preserving realism. The method produces temporally consistent scenario rollouts from the perspectives different actors in the scene, supporting long-horizon continuation of driving scenarios. To facilitate training and evaluation, we release a dataset with text annotations aligned to vectorized map structures. Extensive experiments validate that the control adherence and fidelity of ScenarioControl compare favorable to all tested methods across all experiments. Project webpage: https://light.princeton.edu/ScenarioControl

Related papers

• Video2Track: From Real-World Interaction Videos to Steerable Adversarial Closed-Track Testing for Automated Driving Systems [11.663429828510786]
  Video2Track is a framework that transfers real-world interactive driving scenarios from videos into steerable adversarial closed-track testing.<n>The proposed framework can faithfully reproduce representative real-world interaction scenarios and generate executable scenario variants with controllable risk levels and interaction styles.
  arXiv  Detail & Related papers  (2026-08-12T03:07:00Z)
• End-to-end Conditional Diffusion for Realistic and Controllable Visual Traffic Scenario Generation [59.188006170485046]
  E2E-CDiff is an end-to-end conditional diffusion framework for controllable and realistic scenario generation.<n>Differentiable guidance regulates speed, enforces drivable-area compliance, and supports collision-avoidance or collision-seeking behaviors.<n>E2E-CDiff also performs competitively as a learning-based ego planner.
  arXiv  Detail & Related papers  (2026-07-21T02:13:25Z)
• LLM-based Realistic Safety-Critical Driving Video Generation [4.537331974356809]
  We propose a framework that automatically synthesizes driving scenarios within the CARLA simulator.<n>The framework has flexibility in scenario scripting, efficient code-based control of traffic participants, and enforcement of realistic physical dynamics.<n>Our approach enables controllable scenario generation and facilitates the creation of rare but critical edge cases.
  arXiv  Detail & Related papers  (2025-07-02T00:45:19Z)
• X-Scene: Large-Scale Driving Scene Generation with High Fidelity and Flexible Controllability [49.4647778989539]
  X-Scene is a novel framework for large-scale driving scene generation.<n>It achieves both geometric intricacy and appearance fidelity, while offering flexible controllability.<n>X-Scene significantly advances controllability and fidelity for large-scale driving scene generation.
  arXiv  Detail & Related papers  (2025-06-16T14:43:18Z)
• Compass Control: Multi Object Orientation Control for Text-to-Image Generation [24.4172525865806]
  Existing approaches for controlling text-to-image diffusion models, while powerful, do not allow for explicit 3D object-centric control.<n>We address the problem of multi-object orientation control in text-to-image diffusion models.<n>This enables the generation of diverse multi-object scenes with precise orientation control for each object.
  arXiv  Detail & Related papers  (2025-04-09T10:15:15Z)
• Generating Multimodal Driving Scenes via Next-Scene Prediction [24.84840824118813]
  Generative models in Autonomous Driving (AD) enable diverse scene creation, yet existing methods fall short by only capturing a limited range of modalities.<n>We introduce a multimodal generation framework that incorporates four major data modalities, including a novel addition of map modality.<n>Our framework effectively generates complex, realistic driving scenes over extended sequences, ensuring multimodal consistency and offering fine-grained control over scene elements.
  arXiv  Detail & Related papers  (2025-03-19T07:20:16Z)
• Seeing Beyond Views: Multi-View Driving Scene Video Generation with Holistic Attention [61.3281618482513]
  We present CogDriving, a novel network designed for synthesizing high-quality multi-view driving videos.<n>CogDriving leverages a Diffusion Transformer architecture with holistic-4D attention modules, enabling simultaneous associations across the dimensions.<n>CogDriving demonstrates strong performance on the nuScenes validation set, achieving an FVD score of 37.8, highlighting its ability to generate realistic driving videos.
  arXiv  Detail & Related papers  (2024-12-04T18:02:49Z)
• Generating Compositional Scenes via Text-to-image RGBA Instance Generation [82.63805151691024]
  Text-to-image diffusion generative models can generate high quality images at the cost of tedious prompt engineering. We propose a novel multi-stage generation paradigm that is designed for fine-grained control, flexibility and interactivity. Our experiments show that our RGBA diffusion model is capable of generating diverse and high quality instances with precise control over object attributes.
  arXiv  Detail & Related papers  (2024-11-16T23:44:14Z)
• Fine-grained Controllable Video Generation via Object Appearance and Context [74.23066823064575]
  We propose fine-grained controllable video generation (FACTOR) to achieve detailed control. FACTOR aims to control objects' appearances and context, including their location and category. Our method achieves controllability of object appearances without finetuning, which reduces the per-subject optimization efforts for the users.
  arXiv  Detail & Related papers  (2023-12-05T17:47:33Z)
• SceneChecker: Boosting Scenario Verification using Symmetry Abstractions [3.8995911009078816]
  SceneChecker is a tool for verifying scenarios involving vehicles executing complex plans in large cluttered workspaces. SceneChecker shows 20x speedup in verification time, even while using those very tools as reachability subroutines.
  arXiv  Detail & Related papers  (2020-11-21T03:18:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.