A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems

• URL: http://arxiv.org/abs/2203.12138v1
• Date: Wed, 23 Mar 2022 02:10:30 GMT
• Title: A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems
• Authors: Dmytro Humeniuk, Foutse Khomh, Giuliano Antoniol
• Abstract summary: We design a search based framework, named AmbieGen, for generating diverse fault revealing test scenarios for autonomous cyber physical systems. We evaluate AmbieGen on three scenario generation case studies, namely a smart-thermostat, a robot obstacle avoidance system, and a vehicle lane keeping assist system.
• Score: 12.93632948681342
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Many modern cyber physical systems incorporate computer vision technologies, complex sensors and advanced control software, allowing them to interact with the environment autonomously. Testing such systems poses numerous challenges: not only should the system inputs be varied, but also the surrounding environment should be accounted for. A number of tools have been developed to test the system model for the possible inputs falsifying its requirements. However, they are not directly applicable to autonomous cyber physical systems, as the inputs to their models are generated while operating in a virtual environment. In this paper, we aim to design a search based framework, named AmbieGen, for generating diverse fault revealing test scenarios for autonomous cyber physical systems. The scenarios represent an environment in which an autonomous agent operates. The framework should be applicable to generating different types of environments. To generate the test scenarios, we leverage the NSGA II algorithm with two objectives. The first objective evaluates the deviation of the observed system behaviour from its expected behaviour. The second objective is the test case diversity, calculated as a Jaccard distance with a reference test case. We evaluate AmbieGen on three scenario generation case studies, namely a smart-thermostat, a robot obstacle avoidance system, and a vehicle lane keeping assist system. We compared three configurations of AmbieGen: based on a single objective genetic algorithm, multi objective, and random search. Both single and multi objective configurations outperform the random search. Multi objective configuration can find the individuals of the same quality as the single objective, producing more unique test scenarios in the same time budget.

Related papers

• A Roadmap for Simulation-Based Testing of Autonomous Cyber-Physical Systems: Challenges and Future Direction [5.742965094549775]
  This paper pioneers a strategic roadmap for simulation-based testing of autonomous systems. Our paper discusses the relevant challenges and obstacles of ACPSs, focusing on test automation and quality assurance.
  arXiv  Detail & Related papers  (2024-05-02T07:42:33Z)
• First-principles Based 3D Virtual Simulation Testing for Discovering SOTIF Corner Cases of Autonomous Driving [5.582213904792781]
  This paper proposes a first-principles based sensor modeling and environment interaction scheme, and integrates it into CARLA simulator. A meta-heuristic algorithm is designed based on several empirical insights, which guide both seed scenarios and mutations. Under identical simulation setups, our algorithm discovers about four times as many corner cases as compared to state-of-the-art work.
  arXiv  Detail & Related papers  (2024-01-22T12:02:32Z)
• DARTH: Holistic Test-time Adaptation for Multiple Object Tracking [87.72019733473562]
  Multiple object tracking (MOT) is a fundamental component of perception systems for autonomous driving. Despite the urge of safety in driving systems, no solution to the MOT adaptation problem to domain shift in test-time conditions has ever been proposed. We introduce DARTH, a holistic test-time adaptation framework for MOT.
  arXiv  Detail & Related papers  (2023-10-03T10:10:42Z)
• Interactive System-wise Anomaly Detection [66.3766756452743]
  Anomaly detection plays a fundamental role in various applications. It is challenging for existing methods to handle the scenarios where the instances are systems whose characteristics are not readily observed as data. We develop an end-to-end approach which includes an encoder-decoder module that learns system embeddings.
  arXiv  Detail & Related papers  (2023-04-21T02:20:24Z)
• AmbieGen: A Search-based Framework for Autonomous Systems Testing [12.93632948681342]
  AmbieGen is a search-based test case generation framework for autonomous systems. It supports test case generation for autonomous robots and autonomous car lane keeping assist systems.
  arXiv  Detail & Related papers  (2023-01-01T23:42:32Z)
• Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systems [0.6690874707758508]
  Deep Neural Networks (DNNs) have been widely used to perform real-world tasks in cyber-physical systems such as Autonomous Diving Systems (ADS) Ensuring the correct behavior of such DNN-Enabled Systems (DES) is a crucial topic. Online testing is one of the promising modes for testing such systems with their application environments (simulated or real) in a closed loop. We present MORLOT, a novel online testing approach to address these challenges by combining Reinforcement Learning (RL) and many-objective search.
  arXiv  Detail & Related papers  (2022-10-27T13:53:37Z)
• Learning to Walk Autonomously via Reset-Free Quality-Diversity [73.08073762433376]
  Quality-Diversity algorithms can discover large and complex behavioural repertoires consisting of both diverse and high-performing skills. Existing QD algorithms need large numbers of evaluations as well as episodic resets, which require manual human supervision and interventions. This paper proposes Reset-Free Quality-Diversity optimization (RF-QD) as a step towards autonomous learning for robotics in open-ended environments.
  arXiv  Detail & Related papers  (2022-04-07T14:07:51Z)
• MetaGraspNet: A Large-Scale Benchmark Dataset for Vision-driven Robotic Grasping via Physics-based Metaverse Synthesis [78.26022688167133]
  We present a large-scale benchmark dataset for vision-driven robotic grasping via physics-based metaverse synthesis. The proposed dataset contains 100,000 images and 25 different object types. We also propose a new layout-weighted performance metric alongside the dataset for evaluating object detection and segmentation performance.
  arXiv  Detail & Related papers  (2021-12-29T17:23:24Z)
• Attention-based Neural Network for Driving Environment Complexity Perception [123.93460670568554]
  This paper proposes a novel attention-based neural network model to predict the complexity level of the surrounding driving environment. It consists of a Yolo-v3 object detection algorithm, a heat map generation algorithm, CNN-based feature extractors, and attention-based feature extractors. The proposed attention-based network achieves 91.22% average classification accuracy to classify the surrounding environment complexity.
  arXiv  Detail & Related papers  (2021-06-21T17:27:11Z)
• Data Driven Testing of Cyber Physical Systems [12.93632948681342]
  We propose an approach to automatically generate fault-revealing test cases for CPS. Data collected from an application managing a smart building have been used to learn models of the environment.
  arXiv  Detail & Related papers  (2021-02-23T04:55:10Z)
• Rearrangement: A Challenge for Embodied AI [229.8891614821016]
  We describe a framework for research and evaluation in Embodied AI. Our proposal is based on a canonical task: Rearrangement. We present experimental testbeds of rearrangement scenarios in four different simulation environments.
  arXiv  Detail & Related papers  (2020-11-03T19:42:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.