SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems

• URL: http://arxiv.org/abs/2412.13802v1
• Date: Wed, 18 Dec 2024 12:49:57 GMT
• Title: SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems
• Authors: Huiwen Yang, Yu Zhou, Taolue Chen,
• Abstract summary: SimADFuzz is a novel framework designed to generate high-quality scenarios that reveal violations in autonomous driving systems. SimADFuzz employs violation prediction models, which evaluate the likelihood of ADS violations, to optimize scenario selection. Comprehensive experiments demonstrate that SimADFuzz outperforms state-of-the-art fuzzers by identifying 32 more unique violations.
• Score: 5.738863204900633
• License:
• Abstract: Autonomous driving systems (ADS) have achieved remarkable progress in recent years. However, ensuring their safety and reliability remains a critical challenge due to the complexity and uncertainty of driving scenarios. In this paper, we focus on simulation testing for ADS, where generating diverse and effective testing scenarios is a central task. Existing fuzz testing methods face limitations, such as overlooking the temporal and spatial dynamics of scenarios and failing to leverage simulation feedback (e.g., speed, acceleration and heading) to guide scenario selection and mutation. To address these issues, we propose SimADFuzz, a novel framework designed to generate high-quality scenarios that reveal violations in ADS behavior. Specifically, SimADFuzz employs violation prediction models, which evaluate the likelihood of ADS violations, to optimize scenario selection. Moreover, SimADFuzz proposes distance-guided mutation strategies to enhance interactions among vehicles in offspring scenarios, thereby triggering more edge-case behaviors of vehicles. Comprehensive experiments demonstrate that SimADFuzz outperforms state-of-the-art fuzzers by identifying 32 more unique violations, including 4 reproducible cases of vehicle-vehicle and vehicle-pedestrian collisions. These results demonstrate SimADFuzz's effectiveness in enhancing the robustness and safety of autonomous driving systems.

Related papers

• Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving [65.61999354218628]
  We take the first step toward designing black-box adversarial attacks specifically targeting vision-language models (VLMs) in autonomous driving systems. We propose Cascading Adversarial Disruption (CAD), which targets low-level reasoning breakdown by generating and injecting semantics. We present Risky Scene Induction, which addresses dynamic adaptation by leveraging a surrogate VLM to understand and construct high-level risky scenarios.
  arXiv  Detail & Related papers  (2025-01-23T11:10:02Z)
• Generating Critical Scenarios for Testing Automated Driving Systems [5.975915967339764]
  AVASTRA is a Reinforcement Learning-based approach to generate realistic critical scenarios for testing Autonomous Driving System. Results show AVASTRA's ability to outperform the state-of-the-art approach by generating 30% to 115% more collision scenarios.
  arXiv  Detail & Related papers  (2024-12-03T16:59:30Z)
• ICSFuzz: Collision Detector Bug Discovery in Autonomous Driving Simulators [11.343198884451166]
  This paper aims to systematically discover ignored collision scenarios to improve the reliability of autonomous driving simulators. We present ICSFuzz, a black-box fuzzing approach to discover ignored collision scenarios efficiently. We compare ICSFuzz with DriveFuzz, a state-of-the-art simulation-based ADS testing method, by replacing its oracle with our ignored-collision-aware oracle.
  arXiv  Detail & Related papers  (2024-08-11T04:48:54Z)
• PAFOT: A Position-Based Approach for Finding Optimal Tests of Autonomous Vehicles [4.243926243206826]
  This paper proposes PAFOT, a position-based approach testing framework. PAFOT generates adversarial driving scenarios to expose safety violations of Automated Driving Systems. Experiments show PAFOT can effectively generate safety-critical scenarios to crash ADSs and is able to find collisions in a short simulation time.
  arXiv  Detail & Related papers  (2024-05-06T10:04:40Z)
• SAFE-SIM: Safety-Critical Closed-Loop Traffic Simulation with Diffusion-Controllable Adversaries [94.84458417662407]
  We introduce SAFE-SIM, a controllable closed-loop safety-critical simulation framework. Our approach yields two distinct advantages: 1) generating realistic long-tail safety-critical scenarios that closely reflect real-world conditions, and 2) providing controllable adversarial behavior for more comprehensive and interactive evaluations. We validate our framework empirically using the nuScenes and nuPlan datasets across multiple planners, demonstrating improvements in both realism and controllability.
  arXiv  Detail & Related papers  (2023-12-31T04:14:43Z)
• DARTH: Holistic Test-time Adaptation for Multiple Object Tracking [87.72019733473562]
  Multiple object tracking (MOT) is a fundamental component of perception systems for autonomous driving. Despite the urge of safety in driving systems, no solution to the MOT adaptation problem to domain shift in test-time conditions has ever been proposed. We introduce DARTH, a holistic test-time adaptation framework for MOT.
  arXiv  Detail & Related papers  (2023-10-03T10:10:42Z)
• ReMAV: Reward Modeling of Autonomous Vehicles for Finding Likely Failure Events [1.84926694477846]
  We propose a black-box testing framework that uses offline trajectories first to analyze the existing behavior of autonomous vehicles. Our experiment shows an increase in 35, 23, 48, and 50% in the occurrences of vehicle collision, road object collision, pedestrian collision, and offroad steering events.
  arXiv  Detail & Related papers  (2023-08-28T13:09:00Z)
• BehAVExplor: Behavior Diversity Guided Testing for Autonomous Driving Systems [27.223488110349567]
  Testing autonomous driving systems (ADSs) is a critical task for ensuring their reliability and safety. Existing methods mainly focus on searching for safety violations while the diversity of the generated test cases is ignored. We present a novel behavior-guided fuzzing technique (BehAVExplor) to explore the different behaviors of the ego vehicle and detect diverse violations.
  arXiv  Detail & Related papers  (2023-07-14T17:24:39Z)
• Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles [86.9067793493874]
  We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
  arXiv  Detail & Related papers  (2021-03-12T17:00:23Z)
• Testing the Safety of Self-driving Vehicles by Simulating Perception and Prediction [88.0416857308144]
  We propose an alternative to sensor simulation, as sensor simulation is expensive and has large domain gaps. We directly simulate the outputs of the self-driving vehicle's perception and prediction system, enabling realistic motion planning testing.
  arXiv  Detail & Related papers  (2020-08-13T17:20:02Z)
• Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts? [104.04999499189402]
  Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment. We propose an uncertainty-aware planning method, called emphrobust imitative planning (RIP) Our method can detect and recover from some distribution shifts, reducing the overconfident and catastrophic extrapolations in OOD scenes. We introduce an autonomous car novel-scene benchmark, textttCARNOVEL, to evaluate the robustness of driving agents to a suite of tasks with distribution shifts.
  arXiv  Detail & Related papers  (2020-06-26T11:07:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.