Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to
the Real World
- URL: http://arxiv.org/abs/2003.07739v2
- Date: Sun, 12 Jul 2020 15:00:40 GMT
- Title: Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to
the Real World
- Authors: Daniel J. Fremont, Edward Kim, Yash Vardhan Pant, Sanjit A. Seshia,
Atul Acharya, Xantha Bruso, Paul Wells, Steve Lemke, Qiang Lu, Shalin Mehta
- Abstract summary: We present a new approach to automated scenario-based testing of the safety of autonomous vehicles.
Our approach is based on formal methods, combining formal specification of scenarios and safety properties.
- Score: 8.498542964344987
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present a new approach to automated scenario-based testing of the safety
of autonomous vehicles, especially those using advanced artificial
intelligence-based components, spanning both simulation-based evaluation as
well as testing in the real world. Our approach is based on formal methods,
combining formal specification of scenarios and safety properties, algorithmic
test case generation using formal simulation, test case selection for track
testing, executing test cases on the track, and analyzing the resulting data.
Experiments with a real autonomous vehicle at an industrial testing facility
support our hypotheses that (i) formal simulation can be effective at
identifying test cases to run on the track, and (ii) the gap between simulated
and real worlds can be systematically evaluated and bridged.
Related papers
- Correlation of Software-in-the-Loop Simulation with Physical Testing for Autonomous Driving [0.0]
This paper presents a case study on the validation of an in-house developed SIL simulation toolchain.
To align the test track runs with the SIL simulations, a synchronization approach is proposed.
Preliminary results are presented to demonstrate the effectiveness of the proposed validation process.
arXiv Detail & Related papers (2024-06-05T08:11:10Z) - Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous
Driving Research [76.93956925360638]
Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes.
It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training.
We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
arXiv Detail & Related papers (2023-10-12T20:49:15Z) - Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing [10.518360486008964]
We introduce the notion of digital siblings, a multi-simulator approach that tests a given autonomous vehicle on multiple general-purpose simulators.
We empirically compare such a multi-simulator approach against a digital twin of a physical scaled autonomous vehicle on a large set of test cases.
Our empirical evaluation shows that the ensemble failure predictor by the digital siblings is superior to each individual simulator at predicting the failures of the digital twin.
arXiv Detail & Related papers (2023-05-14T04:10:56Z) - Synthetic Data-Based Simulators for Recommender Systems: A Survey [55.60116686945561]
This survey aims at providing a comprehensive overview of the recent trends in the field of modeling and simulation.
We start with the motivation behind the development of frameworks implementing the simulations -- simulators.
We provide a new consistent classification of existing simulators based on their functionality, approbation, and industrial effectiveness.
arXiv Detail & Related papers (2022-06-22T19:33:21Z) - Efficient and Effective Generation of Test Cases for Pedestrian
Detection -- Search-based Software Testing of Baidu Apollo in SVL [14.482670650074885]
This paper presents a study on testing pedestrian detection and emergency braking system of the Baidu Apollo autonomous driving platform within the SVL simulator.
We propose an evolutionary automated test generation technique that generates failure-revealing scenarios for Apollo in the SVL environment.
In order to demonstrate the efficiency and effectiveness of our approach, we also report the results from a baseline random generation technique.
arXiv Detail & Related papers (2021-09-16T13:11:53Z) - Generating and Characterizing Scenarios for Safety Testing of Autonomous
Vehicles [86.9067793493874]
We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator.
We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project.
We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
arXiv Detail & Related papers (2021-03-12T17:00:23Z) - Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in
Two Industry-Grade Automotive Simulators [13.386879259549305]
We show that SBST can be used to effectively and efficiently generate critical test scenarios in two simulators.
We find that executing the same test scenarios in the two simulators leads to notable differences in the details of the test outputs.
arXiv Detail & Related papers (2020-12-12T14:00:33Z) - A User's Guide to Calibrating Robotics Simulators [54.85241102329546]
This paper proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world.
We conduct experiments on a wide range of well known simulated environments to characterize and offer insights into the performance of different algorithms.
Our analysis can be useful for practitioners working in this area and can help make informed choices about the behavior and main properties of sim-to-real algorithms.
arXiv Detail & Related papers (2020-11-17T22:24:26Z) - Testing the Safety of Self-driving Vehicles by Simulating Perception and
Prediction [88.0416857308144]
We propose an alternative to sensor simulation, as sensor simulation is expensive and has large domain gaps.
We directly simulate the outputs of the self-driving vehicle's perception and prediction system, enabling realistic motion planning testing.
arXiv Detail & Related papers (2020-08-13T17:20:02Z) - From Simulation to Real World Maneuver Execution using Deep
Reinforcement Learning [69.23334811890919]
Deep Reinforcement Learning has proved to be able to solve many control tasks in different fields, but the behavior of these systems is not always as expected when deployed in real-world scenarios.
This is mainly due to the lack of domain adaptation between simulated and real-world data together with the absence of distinction between train and test datasets.
We present a system based on multiple environments in which agents are trained simultaneously, evaluating the behavior of the model in different scenarios.
arXiv Detail & Related papers (2020-05-13T14:22:20Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.