Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in
Two Industry-Grade Automotive Simulators
- URL: http://arxiv.org/abs/2012.06822v2
- Date: Thu, 28 Jan 2021 08:55:23 GMT
- Title: Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in
Two Industry-Grade Automotive Simulators
- Authors: Markus Borg, Raja Ben Abdessalem, Shiva Nejati, Francois-Xavier
Jegeden, Donghwan Shin
- Abstract summary: We show that SBST can be used to effectively and efficiently generate critical test scenarios in two simulators.
We find that executing the same test scenarios in the two simulators leads to notable differences in the details of the test outputs.
- Score: 13.386879259549305
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The increasing levels of software- and data-intensive driving automation call
for an evolution of automotive software testing. As a recommended practice of
the Verification and Validation (V&V) process of ISO/PAS 21448, a candidate
standard for safety of the intended functionality for road vehicles,
simulation-based testing has the potential to reduce both risks and costs.
There is a growing body of research on devising test automation techniques
using simulators for Advanced Driver-Assistance Systems (ADAS). However, how
similar are the results if the same test scenarios are executed in different
simulators? We conduct a replication study of applying a Search-Based Software
Testing (SBST) solution to a real-world ADAS (PeVi, a pedestrian vision
detection system) using two different commercial simulators, namely,
TASS/Siemens PreScan and ESI Pro-SiVIC. Based on a minimalistic scene, we
compare critical test scenarios generated using our SBST solution in these two
simulators. We show that SBST can be used to effectively and efficiently
generate critical test scenarios in both simulators, and the test results
obtained from the two simulators can reveal several weaknesses of the ADAS
under test. However, executing the same test scenarios in the two simulators
leads to notable differences in the details of the test outputs, in particular,
related to (1) safety violations revealed by tests, and (2) dynamics of cars
and pedestrians. Based on our findings, we recommend future V&V plans to
include multiple simulators to support robust simulation-based testing and to
base test objectives on measures that are less dependant on the internals of
the simulators.
Related papers
- Correlation of Software-in-the-Loop Simulation with Physical Testing for Autonomous Driving [0.0]
This paper presents a case study on the validation of an in-house developed SIL simulation toolchain.
To align the test track runs with the SIL simulations, a synchronization approach is proposed.
Preliminary results are presented to demonstrate the effectiveness of the proposed validation process.
arXiv Detail & Related papers (2024-06-05T08:11:10Z) - Automated System-level Testing of Unmanned Aerial Systems [2.2249176072603634]
A major requirement of international safety standards is to perform rigorous system-level testing of avionics software systems.
The proposed approach (AITester) utilizes model-based testing and artificial intelligence (AI) techniques to automatically generate, execute, and evaluate various test scenarios.
arXiv Detail & Related papers (2024-03-23T14:47:26Z) - Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous
Driving Research [76.93956925360638]
Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes.
It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training.
We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
arXiv Detail & Related papers (2023-10-12T20:49:15Z) - Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing [10.518360486008964]
We introduce the notion of digital siblings, a multi-simulator approach that tests a given autonomous vehicle on multiple general-purpose simulators.
We empirically compare such a multi-simulator approach against a digital twin of a physical scaled autonomous vehicle on a large set of test cases.
Our empirical evaluation shows that the ensemble failure predictor by the digital siblings is superior to each individual simulator at predicting the failures of the digital twin.
arXiv Detail & Related papers (2023-05-14T04:10:56Z) - Robust Continual Test-time Adaptation: Instance-aware BN and
Prediction-balanced Memory [58.72445309519892]
We present a new test-time adaptation scheme that is robust against non-i.i.d. test data streams.
Our novelty is mainly two-fold: (a) Instance-Aware Batch Normalization (IABN) that corrects normalization for out-of-distribution samples, and (b) Prediction-balanced Reservoir Sampling (PBRS) that simulates i.i.d. data stream from non-i.i.d. stream in a class-balanced manner.
arXiv Detail & Related papers (2022-08-10T03:05:46Z) - Synthetic Data-Based Simulators for Recommender Systems: A Survey [55.60116686945561]
This survey aims at providing a comprehensive overview of the recent trends in the field of modeling and simulation.
We start with the motivation behind the development of frameworks implementing the simulations -- simulators.
We provide a new consistent classification of existing simulators based on their functionality, approbation, and industrial effectiveness.
arXiv Detail & Related papers (2022-06-22T19:33:21Z) - Metaphorical User Simulators for Evaluating Task-oriented Dialogue
Systems [80.77917437785773]
Task-oriented dialogue systems ( TDSs) are assessed mainly in an offline setting or through human evaluation.
We propose a metaphorical user simulator for end-to-end TDS evaluation, where we define a simulator to be metaphorical if it simulates user's analogical thinking in interactions with systems.
We also propose a tester-based evaluation framework to generate variants, i.e., dialogue systems with different capabilities.
arXiv Detail & Related papers (2022-04-02T05:11:03Z) - Generating and Characterizing Scenarios for Safety Testing of Autonomous
Vehicles [86.9067793493874]
We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator.
We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project.
We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
arXiv Detail & Related papers (2021-03-12T17:00:23Z) - Testing the Safety of Self-driving Vehicles by Simulating Perception and
Prediction [88.0416857308144]
We propose an alternative to sensor simulation, as sensor simulation is expensive and has large domain gaps.
We directly simulate the outputs of the self-driving vehicle's perception and prediction system, enabling realistic motion planning testing.
arXiv Detail & Related papers (2020-08-13T17:20:02Z) - Towards Automated Safety Coverage and Testing for Autonomous Vehicles
with Reinforcement Learning [0.3683202928838613]
Validation puts the autonomous vehicle system to the test in scenarios or situations that the system would likely encounter in everyday driving.
We propose using reinforcement learning (RL) to generate failure examples and unexpected traffic situations for the AV software implementation.
arXiv Detail & Related papers (2020-05-22T19:00:38Z) - Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to
the Real World [8.498542964344987]
We present a new approach to automated scenario-based testing of the safety of autonomous vehicles.
Our approach is based on formal methods, combining formal specification of scenarios and safety properties.
arXiv Detail & Related papers (2020-03-17T14:17:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.