Falsification of Cyber-Physical Systems using Bayesian Optimization

• URL: http://arxiv.org/abs/2209.06735v3
• Date: Wed, 12 Feb 2025 09:32:17 GMT
• Title: Falsification of Cyber-Physical Systems using Bayesian Optimization
• Authors: Zahra Ramezani, Kenan Šehić, Luigi Nardi, Knut Åkesson,
• Abstract summary: Cyber-physical systems (CPSs) are often complex and safety-critical, making it both challenging and crucial to ensure that the system's specifications are met. Simulation-based falsification is a practical testing technique for increasing confidence in a CPS's correctness, as it only requires that the system be simulated. This study investigates a sample-efficient approach that learns a surrogate model to capture the relationship between input signal parameterization and specification evaluation.
• Score: 4.164131508933521
• License:
• Abstract: Cyber-physical systems (CPSs) are often complex and safety-critical, making it both challenging and crucial to ensure that the system's specifications are met. Simulation-based falsification is a practical testing technique for increasing confidence in a CPS's correctness, as it only requires that the system be simulated. Reducing the number of computationally intensive simulations needed for falsification is a key concern. In this study, we investigate Bayesian optimization (BO), a sample-efficient approach that learns a surrogate model to capture the relationship between input signal parameterization and specification evaluation. We propose two enhancements to the basic BO for improving falsification: (1) leveraging local surrogate models, and (2) utilizing the user's prior knowledge. Additionally, we address the formulation of acquisition functions for falsification by proposing and evaluating various alternatives. Our benchmark evaluation demonstrates significant improvements when using local surrogate models in BO for falsifying challenging benchmark examples. Incorporating prior knowledge is found to be especially beneficial when the simulation budget is constrained. For some benchmark problems, the choice of acquisition function noticeably impacts the number of simulations required for successful falsification.

Related papers

• Scoring Verifiers: Evaluating Synthetic Verification in Code and Reasoning [59.25951947621526]
  We introduce benchmarks designed to evaluate the impact of synthetic verification methods on assessing solution correctness. We analyze synthetic verification methods in standard, reasoning-based, and reward-based LLMs. Our results show that recent reasoning models significantly improve test case generation and that scaling test cases enhances verification accuracy.
  arXiv  Detail & Related papers  (2025-02-19T15:32:11Z)
• Synergistic Development of Perovskite Memristors and Algorithms for Robust Analog Computing [53.77822620185878]
  We propose a synergistic methodology to concurrently optimize perovskite memristor fabrication and develop robust analog DNNs. We develop "BayesMulti", a training strategy utilizing BO-guided noise injection to improve the resistance of analog DNNs to memristor imperfections. Our integrated approach enables use of analog computing in much deeper and wider networks, achieving up to 100-fold improvements.
  arXiv  Detail & Related papers  (2024-12-03T19:20:08Z)
• Autoformulation of Mathematical Optimization Models Using LLMs [50.030647274271516]
  We develop an automated approach to creating optimization models from natural language descriptions for commercial solvers. We identify the three core challenges of autoformulation: (1) defining the vast, problem-dependent hypothesis space, (2) efficiently searching this space under uncertainty, and (3) evaluating formulation correctness.
  arXiv  Detail & Related papers  (2024-11-03T20:41:38Z)
• Optimizing Falsification for Learning-Based Control Systems: A Multi-Fidelity Bayesian Approach [40.58350379106314]
  falsification problem involves the identification of counterexamples that violate system safety requirements. We propose a multi-fidelity Bayesian optimization falsification framework that harnesses simulators with varying levels of accuracy.
  arXiv  Detail & Related papers  (2024-09-12T14:51:03Z)
• Calibrating Bayesian Learning via Regularization, Confidence Minimization, and Selective Inference [37.82259435084825]
  A well-calibrated AI model must correctly report its accuracy on in-distribution (ID) inputs, while also enabling the detection of out-of-distribution (OOD) inputs. This paper proposes an extension of variational inference (VI)-based Bayesian learning that integrates calibration regularization for improved ID performance.
  arXiv  Detail & Related papers  (2024-04-17T13:08:26Z)
• Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models [88.80146574509195]
  Quantization is a promising approach for reducing memory overhead and accelerating inference. We propose a novel-aware quantization (ZSAQ) framework for the zero-shot quantization of various PLMs.
  arXiv  Detail & Related papers  (2023-10-20T07:09:56Z)
• Physics-Driven ML-Based Modelling for Correcting Inverse Estimation [6.018296524383859]
  This work focuses on detecting and correcting failed state estimations before adopting them in SAE inverse problems. We propose a novel approach, GEESE, to correct it through optimization, aiming at delivering both low error and high efficiency. GEESE is tested on three real-world SAE inverse problems and compared to a number of state-of-the-art optimization/search approaches.
  arXiv  Detail & Related papers  (2023-09-25T09:37:19Z)
• Simulation-to-reality UAV Fault Diagnosis with Deep Learning [20.182411473467656]
  We propose a deep learning model that addresses the simulation-to-reality gap in fault diagnosis of quadrotors. Our proposed approach achieves an accuracy of 96% in detecting propeller faults. This is the first reliable and efficient method for simulation-to-reality fault diagnosis of quadrotor propellers.
  arXiv  Detail & Related papers  (2023-02-09T02:37:48Z)
• Falsification of Learning-Based Controllers through Multi-Fidelity Bayesian Optimization [34.71695000650056]
  We propose a multi-fidelity falsification framework using Bayesian optimization. This method allows us to automatically switch between inexpensive, inaccurate information from a low-fidelity simulator and expensive, accurate information from a high-fidelity simulator.
  arXiv  Detail & Related papers  (2022-12-28T22:48:42Z)
• Exploring validation metrics for offline model-based optimisation with diffusion models [50.404829846182764]
  In model-based optimisation (MBO) we are interested in using machine learning to design candidates that maximise some measure of reward with respect to a black box function called the (ground truth) oracle. While an approximation to the ground oracle can be trained and used in place of it during model validation to measure the mean reward over generated candidates, the evaluation is approximate and vulnerable to adversarial examples. This is encapsulated under our proposed evaluation framework which is also designed to measure extrapolation.
  arXiv  Detail & Related papers  (2022-11-19T16:57:37Z)
• A Stable, Fast, and Fully Automatic Learning Algorithm for Predictive Coding Networks [65.34977803841007]
  Predictive coding networks are neuroscience-inspired models with roots in both Bayesian statistics and neuroscience. We show how by simply changing the temporal scheduling of the update rule for the synaptic weights leads to an algorithm that is much more efficient and stable than the original one.
  arXiv  Detail & Related papers  (2022-11-16T00:11:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.