Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing

• URL: http://arxiv.org/abs/2305.08060v3
• Date: Wed, 09 Oct 2024 09:14:58 GMT
• Title: Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing
• Authors: Matteo Biagiola, Andrea Stocco, Vincenzo Riccio, Paolo Tonella,
• Abstract summary: We introduce the notion of digital siblings, a multi-simulator approach that tests a given autonomous vehicle on multiple general-purpose simulators. We empirically compare such a multi-simulator approach against a digital twin of a physical scaled autonomous vehicle on a large set of test cases. Our empirical evaluation shows that the ensemble failure predictor by the digital siblings is superior to each individual simulator at predicting the failures of the digital twin.
• Score: 10.518360486008964
• License:
• Abstract: Simulation-based testing represents an important step to ensure the reliability of autonomous driving software. In practice, when companies rely on third-party general-purpose simulators, either for in-house or outsourced testing, the generalizability of testing results to real autonomous vehicles is at stake. In this paper, we enhance simulation-based testing by introducing the notion of digital siblings, a multi-simulator approach that tests a given autonomous vehicle on multiple general-purpose simulators built with different technologies, that operate collectively as an ensemble in the testing process. We exemplify our approach on a case study focused on testing the lane-keeping component of an autonomous vehicle. We use two open-source simulators as digital siblings, and we empirically compare such a multi-simulator approach against a digital twin of a physical scaled autonomous vehicle on a large set of test cases. Our approach requires generating and running test cases for each individual simulator, in the form of sequences of road points. Then, test cases are migrated between simulators, using feature maps to characterize the exercised driving conditions. Finally, the joint predicted failure probability is computed, and a failure is reported only in cases of agreement among the siblings. Our empirical evaluation shows that the ensemble failure predictor by the digital siblings is superior to each individual simulator at predicting the failures of the digital twin. We discuss the findings of our case study and detail how our approach can help researchers interested in automated testing of autonomous driving software.

Related papers

• Correlation of Software-in-the-Loop Simulation with Physical Testing for Autonomous Driving [0.0]
  This paper presents a case study on the validation of an in-house developed SIL simulation toolchain. To align the test track runs with the SIL simulations, a synchronization approach is proposed. Preliminary results are presented to demonstrate the effectiveness of the proposed validation process.
  arXiv  Detail & Related papers  (2024-06-05T08:11:10Z)
• Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Research [76.93956925360638]
  Waymax is a new data-driven simulator for autonomous driving in multi-agent scenes. It runs entirely on hardware accelerators such as TPUs/GPUs and supports in-graph simulation for training. We benchmark a suite of popular imitation and reinforcement learning algorithms with ablation studies on different design decisions.
  arXiv  Detail & Related papers  (2023-10-12T20:49:15Z)
• Indy Autonomous Challenge -- Autonomous Race Cars at the Handling Limits [81.22616193933021]
  The team TUM Auton-omous Motorsports will participate in the Indy Autonomous Challenge in Octo-ber 2021. It will benchmark its self-driving software-stack by racing one out of ten autonomous Dallara AV-21 racecars at the Indianapolis Motor Speedway. It is an ideal testing ground for the development of autonomous driving algorithms capable of mastering the most challenging and rare situations.
  arXiv  Detail & Related papers  (2022-02-08T11:55:05Z)
• Towards Optimal Strategies for Training Self-Driving Perception Models in Simulation [98.51313127382937]
  We focus on the use of labels in the synthetic domain alone. Our approach introduces both a way to learn neural-invariant representations and a theoretically inspired view on how to sample the data from the simulator. We showcase our approach on the bird's-eye-view vehicle segmentation task with multi-sensor data.
  arXiv  Detail & Related papers  (2021-11-15T18:37:43Z)
• SilGAN: Generating driving maneuvers for scenario-based software-in-the-loop testing [0.0]
  SilGAN is a deep generative model that eases specification, stimulus generation, and automation of automotive software-in-the-loop testing. The model is trained using data recorded from vehicles in the field.
  arXiv  Detail & Related papers  (2021-07-05T07:17:49Z)
• Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles [86.9067793493874]
  We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
  arXiv  Detail & Related papers  (2021-03-12T17:00:23Z)
• Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators [13.386879259549305]
  We show that SBST can be used to effectively and efficiently generate critical test scenarios in two simulators. We find that executing the same test scenarios in the two simulators leads to notable differences in the details of the test outputs.
  arXiv  Detail & Related papers  (2020-12-12T14:00:33Z)
• A User's Guide to Calibrating Robotics Simulators [54.85241102329546]
  This paper proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world. We conduct experiments on a wide range of well known simulated environments to characterize and offer insights into the performance of different algorithms. Our analysis can be useful for practitioners working in this area and can help make informed choices about the behavior and main properties of sim-to-real algorithms.
  arXiv  Detail & Related papers  (2020-11-17T22:24:26Z)
• Testing the Safety of Self-driving Vehicles by Simulating Perception and Prediction [88.0416857308144]
  We propose an alternative to sensor simulation, as sensor simulation is expensive and has large domain gaps. We directly simulate the outputs of the self-driving vehicle's perception and prediction system, enabling realistic motion planning testing.
  arXiv  Detail & Related papers  (2020-08-13T17:20:02Z)
• Towards Automated Safety Coverage and Testing for Autonomous Vehicles with Reinforcement Learning [0.3683202928838613]
  Validation puts the autonomous vehicle system to the test in scenarios or situations that the system would likely encounter in everyday driving. We propose using reinforcement learning (RL) to generate failure examples and unexpected traffic situations for the AV software implementation.
  arXiv  Detail & Related papers  (2020-05-22T19:00:38Z)
• Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to the Real World [8.498542964344987]
  We present a new approach to automated scenario-based testing of the safety of autonomous vehicles. Our approach is based on formal methods, combining formal specification of scenarios and safety properties.
  arXiv  Detail & Related papers  (2020-03-17T14:17:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.