AmbieGen: A Search-based Framework for Autonomous Systems Testing
- URL: http://arxiv.org/abs/2301.01234v1
- Date: Sun, 1 Jan 2023 23:42:32 GMT
- Title: AmbieGen: A Search-based Framework for Autonomous Systems Testing
- Authors: Dmytro Humeniuk, Foutse Khomh and Giuliano Antoniol
- Abstract summary: AmbieGen is a search-based test case generation framework for autonomous systems.
It supports test case generation for autonomous robots and autonomous car lane keeping assist systems.
- Score: 12.93632948681342
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Thorough testing of safety-critical autonomous systems, such as self-driving
cars, autonomous robots, and drones, is essential for detecting potential
failures before deployment. One crucial testing stage is model-in-the-loop
testing, where the system model is evaluated by executing various scenarios in
a simulator. However, the search space of possible parameters defining these
test scenarios is vast, and simulating all combinations is computationally
infeasible. To address this challenge, we introduce AmbieGen, a search-based
test case generation framework for autonomous systems. AmbieGen uses
evolutionary search to identify the most critical scenarios for a given system,
and has a modular architecture that allows for the addition of new systems
under test, algorithms, and search operators. Currently, AmbieGen supports test
case generation for autonomous robots and autonomous car lane keeping assist
systems. In this paper, we provide a high-level overview of the framework's
architecture and demonstrate its practical use cases.
Related papers
- A Roadmap for Simulation-Based Testing of Autonomous Cyber-Physical Systems: Challenges and Future Direction [5.742965094549775]
This paper pioneers a strategic roadmap for simulation-based testing of autonomous systems.
Our paper discusses the relevant challenges and obstacles of ACPSs, focusing on test automation and quality assurance.
arXiv Detail & Related papers (2024-05-02T07:42:33Z) - Automated System-level Testing of Unmanned Aerial Systems [2.2249176072603634]
A major requirement of international safety standards is to perform rigorous system-level testing of avionics software systems.
The proposed approach (AITester) utilizes model-based testing and artificial intelligence (AI) techniques to automatically generate, execute, and evaluate various test scenarios.
arXiv Detail & Related papers (2024-03-23T14:47:26Z) - UniSim: A Neural Closed-Loop Sensor Simulator [76.79818601389992]
We present UniSim, a neural sensor simulator that takes a single recorded log captured by a sensor-equipped vehicle.
UniSim builds neural feature grids to reconstruct both the static background and dynamic actors in the scene.
We incorporate learnable priors for dynamic objects, and leverage a convolutional network to complete unseen regions.
arXiv Detail & Related papers (2023-08-03T17:56:06Z) - A Search-Based Framework for Automatic Generation of Testing
Environments for Cyber-Physical Systems [12.93632948681342]
We design a search based framework, named AmbieGen, for generating diverse fault revealing test scenarios for autonomous cyber physical systems.
We evaluate AmbieGen on three scenario generation case studies, namely a smart-thermostat, a robot obstacle avoidance system, and a vehicle lane keeping assist system.
arXiv Detail & Related papers (2022-03-23T02:10:30Z) - A Survey on Scenario-Based Testing for Automated Driving Systems in
High-Fidelity Simulation [26.10081199009559]
Testing the system on the road is the closest to real-world and desirable approach, but it is incredibly costly.
A popular alternative is to evaluate an ADS's performance in some well-designed challenging scenarios, a.k.a. scenario-based testing.
High-fidelity simulators have been widely used in this setting to maximize flexibility and convenience in testing what-if scenarios.
arXiv Detail & Related papers (2021-12-02T03:41:33Z) - CausalCity: Complex Simulations with Agency for Causal Discovery and
Reasoning [68.74447489372037]
We present a high-fidelity simulation environment that is designed for developing algorithms for causal discovery and counterfactual reasoning.
A core component of our work is to introduce textitagency, such that it is simple to define and create complex scenarios.
We perform experiments with three state-of-the-art methods to create baselines and highlight the affordances of this environment.
arXiv Detail & Related papers (2021-06-25T00:21:41Z) - Generating and Characterizing Scenarios for Safety Testing of Autonomous
Vehicles [86.9067793493874]
We propose efficient mechanisms to characterize and generate testing scenarios using a state-of-the-art driving simulator.
We use our method to characterize real driving data from the Next Generation Simulation (NGSIM) project.
We rank the scenarios by defining metrics based on the complexity of avoiding accidents and provide insights into how the AV could have minimized the probability of incurring an accident.
arXiv Detail & Related papers (2021-03-12T17:00:23Z) - A User's Guide to Calibrating Robotics Simulators [54.85241102329546]
This paper proposes a set of benchmarks and a framework for the study of various algorithms aimed to transfer models and policies learnt in simulation to the real world.
We conduct experiments on a wide range of well known simulated environments to characterize and offer insights into the performance of different algorithms.
Our analysis can be useful for practitioners working in this area and can help make informed choices about the behavior and main properties of sim-to-real algorithms.
arXiv Detail & Related papers (2020-11-17T22:24:26Z) - Integrated Benchmarking and Design for Reproducible and Accessible
Evaluation of Robotic Agents [61.36681529571202]
We describe a new concept for reproducible robotics research that integrates development and benchmarking.
One of the central components of this setup is the Duckietown Autolab, a standardized setup that is itself relatively low-cost and reproducible.
We validate the system by analyzing the repeatability of experiments conducted using the infrastructure and show that there is low variance across different robot hardware and across different remote labs.
arXiv Detail & Related papers (2020-09-09T15:31:29Z) - Testing the Safety of Self-driving Vehicles by Simulating Perception and
Prediction [88.0416857308144]
We propose an alternative to sensor simulation, as sensor simulation is expensive and has large domain gaps.
We directly simulate the outputs of the self-driving vehicle's perception and prediction system, enabling realistic motion planning testing.
arXiv Detail & Related papers (2020-08-13T17:20:02Z) - Formal Scenario-Based Testing of Autonomous Vehicles: From Simulation to
the Real World [8.498542964344987]
We present a new approach to automated scenario-based testing of the safety of autonomous vehicles.
Our approach is based on formal methods, combining formal specification of scenarios and safety properties.
arXiv Detail & Related papers (2020-03-17T14:17:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.