Identifying and Explaining Safety-critical Scenarios for Autonomous Vehicles via Key Features

• URL: http://arxiv.org/abs/2212.07566v2
• Date: Tue, 28 Nov 2023 22:50:34 GMT
• Title: Identifying and Explaining Safety-critical Scenarios for Autonomous Vehicles via Key Features
• Authors: Neelofar, Aldeida Aleti
• Abstract summary: This paper uses Instance Space Analysis (ISA) to identify the significant features of test scenarios that affect their ability to reveal the unsafe behaviour of AVs. ISA identifies the features that best differentiate safety-critical scenarios from normal driving and visualises the impact of these features on test scenario outcomes (safe/unsafe) in 2D. To test the predictive ability of the identified features, we train five Machine Learning classifiers to classify test scenarios as safe or unsafe.
• Score: 5.634825161148484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ensuring the safety of autonomous vehicles (AVs) is of utmost importance and testing them in simulated environments is a safer option than conducting in-field operational tests. However, generating an exhaustive test suite to identify critical test scenarios is computationally expensive as the representation of each test is complex and contains various dynamic and static features, such as the AV under test, road participants (vehicles, pedestrians, and static obstacles), environmental factors (weather and light), and the road's structural features (lanes, turns, road speed, etc.). In this paper, we present a systematic technique that uses Instance Space Analysis (ISA) to identify the significant features of test scenarios that affect their ability to reveal the unsafe behaviour of AVs. ISA identifies the features that best differentiate safety-critical scenarios from normal driving and visualises the impact of these features on test scenario outcomes (safe/unsafe) in 2D. This visualization helps to identify untested regions of the instance space and provides an indicator of the quality of the test suite in terms of the percentage of feature space covered by testing. To test the predictive ability of the identified features, we train five Machine Learning classifiers to classify test scenarios as safe or unsafe. The high precision, recall, and F1 scores indicate that our proposed approach is effective in predicting the outcome of a test scenario without executing it and can be used for test generation, selection, and prioritization.

Related papers

• Rigorous Simulation-based Testing for Autonomous Driving Systems -- Targeting the Achilles' Heel of Four Open Autopilots [6.229766691427486]
  We propose a rigorous test method based on breaking down scenarios into simple ones. We generate test cases for critical configurations that place the vehicle under test in critical situations. Test cases reveal major defects in Apollo, Autoware, and the Carla and LGSVL autopilots.
  arXiv  Detail & Related papers  (2024-05-27T08:06:21Z)
• Diversity-guided Search Exploration for Self-driving Cars Test Generation through Frenet Space Encoding [4.135985106933988]
  The rise of self-driving cars (SDCs) presents important safety challenges to address in dynamic environments. While field testing is essential, current methods lack diversity in assessing critical SDC scenarios. We show that the likelihood of leading to an out-of-bound condition can be learned by the deep-learning vanilla transformer model.
  arXiv  Detail & Related papers  (2024-01-26T06:57:00Z)
• ASSERT: Automated Safety Scenario Red Teaming for Evaluating the Robustness of Large Language Models [65.79770974145983]
  ASSERT, Automated Safety Scenario Red Teaming, consists of three methods -- semantically aligned augmentation, target bootstrapping, and adversarial knowledge injection. We partition our prompts into four safety domains for a fine-grained analysis of how the domain affects model performance. We find statistically significant performance differences of up to 11% in absolute classification accuracy among semantically related scenarios and error rates of up to 19% absolute error in zero-shot adversarial settings.
  arXiv  Detail & Related papers  (2023-10-14T17:10:28Z)
• DARTH: Holistic Test-time Adaptation for Multiple Object Tracking [87.72019733473562]
  Multiple object tracking (MOT) is a fundamental component of perception systems for autonomous driving. Despite the urge of safety in driving systems, no solution to the MOT adaptation problem to domain shift in test-time conditions has ever been proposed. We introduce DARTH, a holistic test-time adaptation framework for MOT.
  arXiv  Detail & Related papers  (2023-10-03T10:10:42Z)
• A Requirements-Driven Platform for Validating Field Operations of Small Uncrewed Aerial Vehicles [48.67061953896227]
  DroneReqValidator (DRV) allows sUAS developers to define the operating context, configure multi-sUAS mission requirements, specify safety properties, and deploy their own custom sUAS applications in a high-fidelity 3D environment. The DRV Monitoring system collects runtime data from sUAS and the environment, analyzes compliance with safety properties, and captures violations.
  arXiv  Detail & Related papers  (2023-07-01T02:03:49Z)
• CertainNet: Sampling-free Uncertainty Estimation for Object Detection [65.28989536741658]
  Estimating the uncertainty of a neural network plays a fundamental role in safety-critical settings. In this work, we propose a novel sampling-free uncertainty estimation method for object detection. We call it CertainNet, and it is the first to provide separate uncertainties for each output signal: objectness, class, location and size.
  arXiv  Detail & Related papers  (2021-10-04T17:59:31Z)
• ViSTA: a Framework for Virtual Scenario-based Testing of Autonomous Vehicles [2.20200533591633]
  We present ViSTA, a framework for Virtual Scenario-based Testing of Autonomous Vehicles (AV) We describe a comprehensive test case generation approach that facilitates the design of special-purpose scenarios with meaningful parameters. We describe how to automate the execution of test cases, and analyze the performance of the AV under these test cases.
  arXiv  Detail & Related papers  (2021-09-06T15:12:17Z)
• Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles [86.9067793493874]
  We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
  arXiv  Detail & Related papers  (2021-03-12T17:00:23Z)
• Search-based Test-Case Generation by Monitoring Responsibility Safety Rules [2.1270496914042996]
  We propose a method for screening and classifying simulation-based driving test data to be used for training and testing controllers. Our framework is distributed with the publicly available S-TALIRO and Sim-ATAV tools.
  arXiv  Detail & Related papers  (2020-04-25T10:10:11Z)
• Learning Control Barrier Functions from Expert Demonstrations [69.23675822701357]
  We propose a learning based approach to safe controller synthesis based on control barrier functions (CBFs) We analyze an optimization-based approach to learning a CBF that enjoys provable safety guarantees under suitable Lipschitz assumptions on the underlying dynamical system. To the best of our knowledge, these are the first results that learn provably safe control barrier functions from data.
  arXiv  Detail & Related papers  (2020-04-07T12:29:06Z)
• Efficient statistical validation with edge cases to evaluate Highly Automated Vehicles [6.198523595657983]
  The widescale deployment of Autonomous Vehicles seems to be imminent despite many safety challenges that are yet to be resolved. Existing standards focus on deterministic processes where the validation requires only a set of test cases that cover the requirements. This paper presents a new approach to compute the statistical characteristics of a system's behaviour by biasing automatically generated test cases towards the worst case scenarios.
  arXiv  Detail & Related papers  (2020-03-04T04:35:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.