AI-Augmented Metamorphic Testing for Comprehensive Validation of Autonomous Vehicles

• URL: http://arxiv.org/abs/2502.12208v1
• Date: Sun, 16 Feb 2025 23:31:59 GMT
• Title: AI-Augmented Metamorphic Testing for Comprehensive Validation of Autonomous Vehicles
• Authors: Tony Zhang, Burak Kantarci, Umair Siddique,
• Abstract summary: Self-driving cars have the potential to revolutionize transportation, but ensuring their safety remains a significant challenge. Conventional testing methodologies face critical limitations, including the oracle problem determining whether the systems behavior is correct. We propose enhancing Metamorphic Testing (MT) by integrating AI-driven image generation tools, such as Stable Diffusion.
• Score: 7.237068561453082
• License:
• Abstract: Self-driving cars have the potential to revolutionize transportation, but ensuring their safety remains a significant challenge. These systems must navigate a variety of unexpected scenarios on the road, and their complexity poses substantial difficulties for thorough testing. Conventional testing methodologies face critical limitations, including the oracle problem determining whether the systems behavior is correct and the inability to exhaustively recreate a range of situations a self-driving car may encounter. While Metamorphic Testing (MT) offers a partial solution to these challenges, its application is often limited by simplistic modifications to test scenarios. In this position paper, we propose enhancing MT by integrating AI-driven image generation tools, such as Stable Diffusion, to improve testing methodologies. These tools can generate nuanced variations of driving scenarios within the operational design domain (ODD)for example, altering weather conditions, modifying environmental elements, or adjusting lane markings while preserving the critical features necessary for system evaluation. This approach enables reproducible testing, efficient reuse of test criteria, and comprehensive evaluation of a self-driving systems performance across diverse scenarios, thereby addressing key gaps in current testing practices.

Related papers

• SimADFuzz: Simulation-Feedback Fuzz Testing for Autonomous Driving Systems [5.738863204900633]
  SimADFuzz is a novel framework designed to generate high-quality scenarios that reveal violations in autonomous driving systems. SimADFuzz employs violation prediction models, which evaluate the likelihood of ADS violations, to optimize scenario selection. Comprehensive experiments demonstrate that SimADFuzz outperforms state-of-the-art fuzzers by identifying 32 more unique violations.
  arXiv  Detail & Related papers  (2024-12-18T12:49:57Z)
• The RoboDrive Challenge: Drive Anytime Anywhere in Any Condition [136.32656319458158]
  The 2024 RoboDrive Challenge was crafted to propel the development of driving perception technologies. This year's challenge consisted of five distinct tracks and attracted 140 registered teams from 93 institutes across 11 countries. The competition culminated in 15 top-performing solutions.
  arXiv  Detail & Related papers  (2024-05-14T17:59:57Z)
• DARTH: Holistic Test-time Adaptation for Multiple Object Tracking [87.72019733473562]
  Multiple object tracking (MOT) is a fundamental component of perception systems for autonomous driving. Despite the urge of safety in driving systems, no solution to the MOT adaptation problem to domain shift in test-time conditions has ever been proposed. We introduce DARTH, a holistic test-time adaptation framework for MOT.
  arXiv  Detail & Related papers  (2023-10-03T10:10:42Z)
• Unsupervised Domain Adaptation for Self-Driving from Past Traversal Features [69.47588461101925]
  We propose a method to adapt 3D object detectors to new driving environments. Our approach enhances LiDAR-based detection models using spatial quantized historical features. Experiments on real-world datasets demonstrate significant improvements.
  arXiv  Detail & Related papers  (2023-09-21T15:00:31Z)
• Adaptive Failure Search Using Critical States from Domain Experts [9.93890332477992]
  Failure search may be done through logging substantial vehicle miles in either simulation or real world testing. AST is one such method that poses the problem of failure search as a Markov decision process. We show that the incorporation of critical states into the AST framework generates failure scenarios with increased safety violations.
  arXiv  Detail & Related papers  (2023-04-01T18:14:41Z)
• Identifying and Explaining Safety-critical Scenarios for Autonomous Vehicles via Key Features [5.634825161148484]
  This paper uses Instance Space Analysis (ISA) to identify the significant features of test scenarios that affect their ability to reveal the unsafe behaviour of AVs. ISA identifies the features that best differentiate safety-critical scenarios from normal driving and visualises the impact of these features on test scenario outcomes (safe/unsafe) in 2D. To test the predictive ability of the identified features, we train five Machine Learning classifiers to classify test scenarios as safe or unsafe.
  arXiv  Detail & Related papers  (2022-12-15T00:52:47Z)
• Learning energy-efficient driving behaviors by imitating experts [75.12960180185105]
  This paper examines the role of imitation learning in bridging the gap between control strategies and realistic limitations in communication and sensing. We show that imitation learning can succeed in deriving policies that, if adopted by 5% of vehicles, may boost the energy-efficiency of networks with varying traffic conditions by 15% using only local observations.
  arXiv  Detail & Related papers  (2022-06-28T17:08:31Z)
• Generating and Characterizing Scenarios for Safety Testing of Autonomous Vehicles [86.9067793493874]
  We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator. We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project. We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
  arXiv  Detail & Related papers  (2021-03-12T17:00:23Z)
• Can Autonomous Vehicles Identify, Recover From, and Adapt to Distribution Shifts? [104.04999499189402]
  Out-of-training-distribution (OOD) scenarios are a common challenge of learning agents at deployment. We propose an uncertainty-aware planning method, called emphrobust imitative planning (RIP) Our method can detect and recover from some distribution shifts, reducing the overconfident and catastrophic extrapolations in OOD scenes. We introduce an autonomous car novel-scene benchmark, textttCARNOVEL, to evaluate the robustness of driving agents to a suite of tasks with distribution shifts.
  arXiv  Detail & Related papers  (2020-06-26T11:07:32Z)
• Towards Automated Safety Coverage and Testing for Autonomous Vehicles with Reinforcement Learning [0.3683202928838613]
  Validation puts the autonomous vehicle system to the test in scenarios or situations that the system would likely encounter in everyday driving. We propose using reinforcement learning (RL) to generate failure examples and unexpected traffic situations for the AV software implementation.
  arXiv  Detail & Related papers  (2020-05-22T19:00:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.