SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout

• URL: http://arxiv.org/abs/2412.12129v1
• Date: Thu, 05 Dec 2024 18:06:53 GMT
• Title: SceneDiffuser: Efficient and Controllable Driving Simulation Initialization and Rollout
• Authors: Chiyu Max Jiang, Yijing Bai, Andre Cornman, Christopher Davis, Xiukun Huang, Hong Jeon, Sakshum Kulshrestha, John Lambert, Shuangyu Li, Xuanyu Zhou, Carlos Fuertes, Chang Yuan, Mingxing Tan, Yin Zhou, Dragomir Anguelov,
• Abstract summary: Realistic and interactive scene simulation is a key prerequisite for autonomous vehicle (AV) development. We present SceneDiffuser, a scene-level diffusion prior designed for traffic simulation. Novel diffusion denoising paradigm amortizes the computational cost of denoising over future simulation steps.
• Score: 30.098214039454927
• License:
• Abstract: Realistic and interactive scene simulation is a key prerequisite for autonomous vehicle (AV) development. In this work, we present SceneDiffuser, a scene-level diffusion prior designed for traffic simulation. It offers a unified framework that addresses two key stages of simulation: scene initialization, which involves generating initial traffic layouts, and scene rollout, which encompasses the closed-loop simulation of agent behaviors. While diffusion models have been proven effective in learning realistic and multimodal agent distributions, several challenges remain, including controllability, maintaining realism in closed-loop simulations, and ensuring inference efficiency. To address these issues, we introduce amortized diffusion for simulation. This novel diffusion denoising paradigm amortizes the computational cost of denoising over future simulation steps, significantly reducing the cost per rollout step (16x less inference steps) while also mitigating closed-loop errors. We further enhance controllability through the introduction of generalized hard constraints, a simple yet effective inference-time constraint mechanism, as well as language-based constrained scene generation via few-shot prompting of a large language model (LLM). Our investigations into model scaling reveal that increased computational resources significantly improve overall simulation realism. We demonstrate the effectiveness of our approach on the Waymo Open Sim Agents Challenge, achieving top open-loop performance and the best closed-loop performance among diffusion models.

Related papers

• Rolling Ahead Diffusion for Traffic Scene Simulation [13.900806577888861]
  Realistic driving simulation requires that NPCs not only mimic natural driving behaviors but also react to the behavior of other simulated agents. Recent developments in diffusion-based scenario generation focus on creating diverse and realistic traffic scenarios. We present a rolling diffusion based traffic scene generation model which mixes the benefits of both methods.
  arXiv  Detail & Related papers  (2025-02-13T18:45:56Z)
• Causal Composition Diffusion Model for Closed-loop Traffic Generation [31.52951126032351]
  We introduce the Causal Compositional Diffusion Model (CCDiff), a structure-guided diffusion framework to address these challenges. We first formulate the learning of controllable and realistic closed-loop simulation as a constrained optimization problem. Then, CCDiff maximizes controllability while adhering to realism by automatically identifying and injecting causal structures directly into the diffusion process.
  arXiv  Detail & Related papers  (2024-12-23T19:20:29Z)
• ACDiT: Interpolating Autoregressive Conditional Modeling and Diffusion Transformer [95.80384464922147]
  Continuous visual generation requires the full-sequence diffusion-based approach. We present ACDiT, an Autoregressive blockwise Conditional Diffusion Transformer. We demonstrate that ACDiT can be seamlessly used in visual understanding tasks despite being trained on the diffusion objective.
  arXiv  Detail & Related papers  (2024-12-10T18:13:20Z)
• Efficient Text-driven Motion Generation via Latent Consistency Training [21.348658259929053]
  We propose a motion latent consistency training framework (MLCT) to solve nonlinear reverse diffusion trajectories. By combining these enhancements, we achieve stable and consistency training in non-pixel modality and latent representation spaces.
  arXiv  Detail & Related papers  (2024-05-05T02:11:57Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research [76.93956925360638]
  Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes. It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training. We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
  arXiv  Detail & Related papers  (2023-10-12T20:49:15Z)
• Layout Sequence Prediction From Noisy Mobile Modality [53.49649231056857]
  Trajectory prediction plays a vital role in understanding pedestrian movement for applications such as autonomous driving and robotics. Current trajectory prediction models depend on long, complete, and accurately observed sequences from visual modalities. We propose LTrajDiff, a novel approach that treats objects obstructed or out of sight as equally important as those with fully visible trajectories.
  arXiv  Detail & Related papers  (2023-10-09T20:32:49Z)
• Language-Guided Traffic Simulation via Scene-Level Diffusion [46.47977644226296]
  We present CTG++, a scene-level conditional diffusion model that can be guided by language instructions. We first propose a scene-level diffusion model equipped with atemporal backbone which generates realistic and controllable traffic. We then harness a large language model (LLM) to convert a users query into a loss function guiding the diffusion model towards query-compliant generation.
  arXiv  Detail & Related papers  (2023-06-10T05:20:30Z)
• When to Update Your Model: Constrained Model-based Reinforcement Learning [50.74369835934703]
  We propose a novel and general theoretical scheme for a non-decreasing performance guarantee of model-based RL (MBRL) Our follow-up derived bounds reveal the relationship between model shifts and performance improvement. A further example demonstrates that learning models from a dynamically-varying number of explorations benefit the eventual returns.
  arXiv  Detail & Related papers  (2022-10-15T17:57:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.