ICSFuzz: Collision Detector Bug Discovery in Autonomous Driving Simulators

• URL: http://arxiv.org/abs/2408.05694v1
• Date: Sun, 11 Aug 2024 04:48:54 GMT
• Title: ICSFuzz: Collision Detector Bug Discovery in Autonomous Driving Simulators
• Authors: Weiwei Fu, Heqing Huang, Yifan Zhang, Ke Zhang, Jin Huang, Wei-Bin Lee, Jianping Wang,
• Abstract summary: This paper aims to systematically discover ignored collision scenarios to improve the reliability of autonomous driving simulators. We present ICSFuzz, a black-box fuzzing approach to discover ignored collision scenarios efficiently. We compare ICSFuzz with DriveFuzz, a state-of-the-art simulation-based ADS testing method, by replacing its oracle with our ignored-collision-aware oracle.
• Score: 11.343198884451166
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the increasing adoption of autonomous vehicles, ensuring the reliability of autonomous driving systems (ADSs) deployed on autonomous vehicles has become a significant concern. Driving simulators have emerged as crucial platforms for testing autonomous driving systems, offering realistic, dynamic, and configurable environments. However, existing simulation-based ADS testers have largely overlooked the reliability of the simulators, potentially leading to overlooked violation scenarios and subsequent safety security risks during real-world deployment. In our investigations, we identified that collision detectors in simulators could fail to detect and report collisions in certain collision scenarios, referred to as ignored collision scenarios. This paper aims to systematically discover ignored collision scenarios to improve the reliability of autonomous driving simulators. To this end, we present ICSFuzz, a black-box fuzzing approach to discover ignored collision scenarios efficiently. Drawing upon the fact that the ignored collision scenarios are a sub-type of collision scenarios, our approach starts with the determined collision scenarios. Following the guidance provided by empirically studied factors contributing to collisions, we selectively mutate arbitrary collision scenarios in a step-wise manner toward the ignored collision scenarios and effectively discover them. We compare ICSFuzz with DriveFuzz, a state-of-the-art simulation-based ADS testing method, by replacing its oracle with our ignored-collision-aware oracle. The evaluation demonstrates that ICSFuzz outperforms DriveFuzz by finding 10-20x more ignored collision scenarios with a 20-70x speedup. All the discovered ignored collisions have been confirmed by developers with one CVE ID assigned.

Related papers

• SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems [5.738863204900633]
  SimADFuzz is a novel framework designed to generate high-quality scenarios that reveal violations in autonomous driving systems. SimADFuzz employs violation prediction models, which evaluate the likelihood of ADS violations, to optimize scenario selection. Comprehensive experiments demonstrate that SimADFuzz outperforms state-of-the-art fuzzers by identifying 32 more unique violations.
  arXiv  Detail & Related papers  (2024-12-18T12:49:57Z)
• Generating Critical Scenarios for Testing Automated Driving Systems [5.975915967339764]
  AVASTRA is a Reinforcement Learning-based approach to generate realistic critical scenarios for testing Autonomous Driving System. Results show AVASTRA's ability to outperform the state-of-the-art approach by generating 30% to 115% more collision scenarios.
  arXiv  Detail & Related papers  (2024-12-03T16:59:30Z)
• Foundation Models for Rapid Autonomy Validation [4.417336418010182]
  A key challenge is that an autonomous vehicle requires testing in every kind of driving scenario it could encounter. We propose the use of a behavior foundation model, specifically a masked autoencoder (MAE), trained to reconstruct driving scenarios.
  arXiv  Detail & Related papers  (2024-10-22T15:32:43Z)
• ReGentS: Real-World Safety-Critical Driving Scenario Generation Made Stable [88.08120417169971]
  Machine learning based autonomous driving systems often face challenges with safety-critical scenarios that are rare in real-world data. This work explores generating safety-critical driving scenarios by modifying complex real-world regular scenarios through trajectory optimization. Our approach addresses unrealistic diverging trajectories and unavoidable collision scenarios that are not useful for training robust planner.
  arXiv  Detail & Related papers  (2024-09-12T08:26:33Z)
• LMM-enhanced Safety-Critical Scenario Generation for Autonomous Driving System Testing From Non-Accident Traffic Videos [22.638869562921133]
  It is paramount to generate a diverse range of safety-critical test scenarios for autonomous driving systems. Some accident-free real-world scenarios can not only lead to misbehaviors in ADSs but also be leveraged for the generation of ADS violations. It is of significant importance to discover safety violations of ADSs from routine traffic scenarios.
  arXiv  Detail & Related papers  (2024-06-16T09:05:56Z)
• PAFOT: A Position-Based Approach for Finding Optimal Tests of Autonomous Vehicles [4.243926243206826]
  This paper proposes PAFOT, a position-based approach testing framework. PAFOT generates adversarial driving scenarios to expose safety violations of Automated Driving Systems. Experiments show PAFOT can effectively generate safety-critical scenarios to crash ADSs and is able to find collisions in a short simulation time.
  arXiv  Detail & Related papers  (2024-05-06T10:04:40Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• UniSim: A Neural Closed-Loop Sensor Simulator [76.79818601389992]
  We present UniSim, a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle. UniSim builds neural feature grids to reconstruct both the static background and dynamic actors in the scene. We incorporate learnable priors for dynamic objects, and leverage a convolutional network to complete unseen regions.
  arXiv  Detail & Related papers  (2023-08-03T17:56:06Z)
• DeepAccident: A Motion and Accident Prediction Benchmark for V2X Autonomous Driving [76.29141888408265]
  We propose a large-scale dataset containing diverse accident scenarios that frequently occur in real-world driving. The proposed DeepAccident dataset includes 57K annotated frames and 285K annotated samples, approximately 7 times more than the large-scale nuScenes dataset.
  arXiv  Detail & Related papers  (2023-04-03T17:37:00Z)
• Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles [86.9067793493874]
  We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
  arXiv  Detail & Related papers  (2021-03-12T17:00:23Z)
• AdvSim: Generating Safety-Critical Scenarios for Self-Driving Vehicles [76.46575807165729]
  We propose AdvSim, an adversarial framework to generate safety-critical scenarios for any LiDAR-based autonomy system. By simulating directly from sensor data, we obtain adversarial scenarios that are safety-critical for the full autonomy stack.
  arXiv  Detail & Related papers  (2021-01-16T23:23:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.