RealGen: Retrieval Augmented Generation for Controllable Traffic
Scenarios
- URL: http://arxiv.org/abs/2312.13303v1
- Date: Tue, 19 Dec 2023 23:11:06 GMT
- Title: RealGen: Retrieval Augmented Generation for Controllable Traffic
Scenarios
- Authors: Wenhao Ding, Yulong Cao, Ding Zhao, Chaowei Xiao, Marco Pavone
- Abstract summary: RealGen is a novel retrieval-based in-context learning framework for traffic scenario generation.
RealGen synthesizes new scenarios by combining behaviors from multiple retrieved examples in a gradient-free way.
This in-context learning framework endows versatile generative capabilities, including the ability to edit scenarios.
- Score: 62.89459646611976
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Simulation plays a crucial role in the development of autonomous vehicles
(AVs) due to the potential risks associated with real-world testing. Although
significant progress has been made in the visual aspects of simulators,
generating complex behavior among agents remains a formidable challenge. It is
not only imperative to ensure realism in the scenarios generated but also
essential to incorporate preferences and conditions to facilitate controllable
generation for AV training and evaluation. Traditional methods, mainly relying
on memorizing the distribution of training datasets, often fall short in
generating unseen scenarios. Inspired by the success of retrieval augmented
generation in large language models, we present RealGen, a novel
retrieval-based in-context learning framework for traffic scenario generation.
RealGen synthesizes new scenarios by combining behaviors from multiple
retrieved examples in a gradient-free way, which may originate from templates
or tagged scenarios. This in-context learning framework endows versatile
generative capabilities, including the ability to edit scenarios, compose
various behaviors, and produce critical scenarios. Evaluations show that
RealGen offers considerable flexibility and controllability, marking a new
direction in the field of controllable traffic scenario generation. Check our
project website for more information: https://realgen.github.io.
Related papers
- SimGen: Simulator-conditioned Driving Scene Generation [50.03358485083602]
SimGen can learn to generate diverse driving scenes by mixing data from the simulator and the real world.
SimGen achieves superior generation quality and diversity while preserving controllability based on the text prompt and the layout pulled from a simulator.
arXiv Detail & Related papers (2024-06-13T17:58:32Z) - CtRL-Sim: Reactive and Controllable Driving Agents with Offline Reinforcement Learning [38.63187494867502]
We propose CtRL-Sim, a method that leverages return-conditioned offline reinforcement learning to efficiently generate controllable traffic agents.
Specifically, we process real-world driving data through a physics-enhanced Nocturne simulator to generate a diverse offline reinforcement learning dataset.
With this dataset, we train a return-conditioned multi-agent behaviour model that allows for fine-grained manipulation of agent behaviours.
arXiv Detail & Related papers (2024-03-29T02:10:19Z) - SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Controllable Adversaries [94.84458417662407]
We introduce SAFE-SIM, a novel diffusion-based controllable closed-loop safety-critical simulation framework.
We develop a novel approach to simulate safety-critical scenarios through an adversarial term in the denoising process.
We validate our framework empirically using the NuScenes dataset, demonstrating improvements in both realism and controllability.
arXiv Detail & Related papers (2023-12-31T04:14:43Z) - Language Conditioned Traffic Generation [37.71751991840586]
LCTGen is a large language model with a transformer-based decoder architecture that selects likely map locations from a dataset of maps.
It produces an initial traffic distribution, as well as the dynamics of each vehicle.
LCTGen outperforms prior work in both unconditional and conditional traffic scene generation in terms of realism and fidelity.
arXiv Detail & Related papers (2023-07-16T05:10:32Z) - NASimEmu: Network Attack Simulator & Emulator for Training Agents
Generalizing to Novel Scenarios [0.0]
NASimEmu is a new framework for training penetration testing agents.
It provides a simulator and an emulator with a shared interface.
We show how to use the framework to train a general agent that transfers into novel, structurally different scenarios.
arXiv Detail & Related papers (2023-05-26T20:19:09Z) - TrajGen: Generating Realistic and Diverse Trajectories with Reactive and
Feasible Agent Behaviors for Autonomous Driving [19.06020265777298]
Existing simulators rely on system-based behavior models for background vehicles, which cannot capture the complex interactive behaviors in real-world scenarios.
We propose TrajGen, a two-stage trajectory generation framework, which can capture more realistic behaviors directly from human demonstration.
In addition, we develop a data-driven simulator I-Sim that can be used to train reinforcement learning models in parallel based on naturalistic driving data.
arXiv Detail & Related papers (2022-03-31T04:48:29Z) - DriveGAN: Towards a Controllable High-Quality Neural Simulation [147.6822288981004]
We introduce a novel high-quality neural simulator referred to as DriveGAN.
DriveGAN achieves controllability by disentangling different components without supervision.
We train DriveGAN on multiple datasets, including 160 hours of real-world driving data.
arXiv Detail & Related papers (2021-04-30T15:30:05Z) - Generating and Characterizing Scenarios for Safety Testing of Autonomous
Vehicles [86.9067793493874]
We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator.
We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project.
We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
arXiv Detail & Related papers (2021-03-12T17:00:23Z) - TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors [74.67698916175614]
We propose TrafficSim, a multi-agent behavior model for realistic traffic simulation.
In particular, we leverage an implicit latent variable model to parameterize a joint actor policy.
We show TrafficSim generates significantly more realistic and diverse traffic scenarios as compared to a diverse set of baselines.
arXiv Detail & Related papers (2021-01-17T00:29:30Z) - Improving the Generalization of End-to-End Driving through Procedural
Generation [35.41368856679809]
We release an open-ended driving simulator called PGDrive to better evaluate and improve generalization of end-to-end driving.
We validate that training with the increasing number of procedurally generated scenes significantly improves the generalization of the agent across scenarios of different traffic densities and road networks.
arXiv Detail & Related papers (2020-12-26T06:23:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.