ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles
- URL: http://arxiv.org/abs/2405.14062v1
- Date: Wed, 22 May 2024 23:21:15 GMT
- Title: ChatScene: Knowledge-Enabled Safety-Critical Scenario Generation for Autonomous Vehicles
- Authors: Jiawei Zhang, Chejian Xu, Bo Li,
- Abstract summary: ChatScene is a Large Language Model (LLM)-based agent that generates safety-critical scenarios for autonomous vehicles.
A key part of our agent is a comprehensive knowledge retrieval component, which efficiently translates specific textual descriptions into corresponding domain-specific code snippets.
- Score: 17.396416459648755
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present ChatScene, a Large Language Model (LLM)-based agent that leverages the capabilities of LLMs to generate safety-critical scenarios for autonomous vehicles. Given unstructured language instructions, the agent first generates textually described traffic scenarios using LLMs. These scenario descriptions are subsequently broken down into several sub-descriptions for specified details such as behaviors and locations of vehicles. The agent then distinctively transforms the textually described sub-scenarios into domain-specific languages, which then generate actual code for prediction and control in simulators, facilitating the creation of diverse and complex scenarios within the CARLA simulation environment. A key part of our agent is a comprehensive knowledge retrieval component, which efficiently translates specific textual descriptions into corresponding domain-specific code snippets by training a knowledge database containing the scenario description and code pairs. Extensive experimental results underscore the efficacy of ChatScene in improving the safety of autonomous vehicles. For instance, the scenarios generated by ChatScene show a 15% increase in collision rates compared to state-of-the-art baselines when tested against different reinforcement learning-based ego vehicles. Furthermore, we show that by using our generated safety-critical scenarios to fine-tune different RL-based autonomous driving models, they can achieve a 9% reduction in collision rates, surpassing current SOTA methods. ChatScene effectively bridges the gap between textual descriptions of traffic scenarios and practical CARLA simulations, providing a unified way to conveniently generate safety-critical scenarios for safety testing and improvement for AVs.
Related papers
- VistaScenario: Interaction Scenario Engineering for Vehicles with Intelligent Systems for Transport Automation [18.897103921181255]
We propose VistaScenario framework to conduct scenario engineering for vehicles with intelligent systems for transport automation.
Based on summarized basic types of vehicle interactions, we slice scenario data stream into segments via scenario evolution tree.
We also propose the scenario metric Graph-DTW based on Graph Tree and Dynamic Time Warping vehicles to conduct scenario comparison and labeling.
arXiv Detail & Related papers (2024-02-12T15:34:04Z) - SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Controllable Adversaries [94.84458417662407]
We introduce SAFE-SIM, a novel diffusion-based controllable closed-loop safety-critical simulation framework.
We develop a novel approach to simulate safety-critical scenarios through an adversarial term in the denoising process.
We validate our framework empirically using the NuScenes dataset, demonstrating improvements in both realism and controllability.
arXiv Detail & Related papers (2023-12-31T04:14:43Z) - RealGen: Retrieval Augmented Generation for Controllable Traffic
Scenarios [62.89459646611976]
RealGen is a novel retrieval-based in-context learning framework for traffic scenario generation.
RealGen synthesizes new scenarios by combining behaviors from multiple retrieved examples in a gradient-free way.
This in-context learning framework endows versatile generative capabilities, including the ability to edit scenarios.
arXiv Detail & Related papers (2023-12-19T23:11:06Z) - Empowering Autonomous Driving with Large Language Models: A Safety Perspective [82.90376711290808]
This paper explores the integration of Large Language Models (LLMs) into Autonomous Driving systems.
LLMs are intelligent decision-makers in behavioral planning, augmented with a safety verifier shield for contextual safety learning.
We present two key studies in a simulated environment: an adaptive LLM-conditioned Model Predictive Control (MPC) and an LLM-enabled interactive behavior planning scheme with a state machine.
arXiv Detail & Related papers (2023-11-28T03:13:09Z) - CAT: Closed-loop Adversarial Training for Safe End-to-End Driving [54.60865656161679]
Adversarial Training (CAT) is a framework for safe end-to-end driving in autonomous vehicles.
Cat aims to continuously improve the safety of driving agents by training the agent on safety-critical scenarios.
Cat can effectively generate adversarial scenarios countering the agent being trained.
arXiv Detail & Related papers (2023-10-19T02:49:31Z) - Generating and Explaining Corner Cases Using Learnt Probabilistic Lane
Graphs [5.309950889075669]
We introduce Probabilistic Lane Graphs (PLGs) to describe a finite set of lane positions and directions in which vehicles might travel.
The structure of PLGs is learnt directly from historic traffic data.
We use reinforcement learning techniques to modify this policy to generate realistic and explainable corner case scenarios.
arXiv Detail & Related papers (2023-08-25T20:17:49Z) - Tree-Based Scenario Classification: A Formal Framework for Coverage
Analysis on Test Drives of Autonomous Vehicles [0.0]
In scenario-based testing, relevant (driving) scenarios are the basis of tests.
We address the open challenges of classifying sets of scenarios and measuring coverage of theses scenarios in recorded test drives.
arXiv Detail & Related papers (2023-07-11T08:30:57Z) - Realistic simulation of users for IT systems in cyber ranges [63.20765930558542]
We instrument each machine by means of an external agent to generate user activity.
This agent combines both deterministic and deep learning based methods to adapt to different environment.
We also propose conditional text generation models to facilitate the creation of conversations and documents.
arXiv Detail & Related papers (2021-11-23T10:53:29Z) - Generating and Characterizing Scenarios for Safety Testing of Autonomous
Vehicles [86.9067793493874]
We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator.
We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project.
We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
arXiv Detail & Related papers (2021-03-12T17:00:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.