Flaky Test Recognition when Testing CPSs Using Hybrid Models

• URL: http://arxiv.org/abs/2608.06535v1
• Date: Thu, 06 Aug 2026 19:39:46 GMT
• Title: Flaky Test Recognition when Testing CPSs Using Hybrid Models
• Abstract summary: Cyber-Physical Systems (CPSs) have many applications, ranging from simple thermostat systems to autonomous driving systems and medical devices.<n>CPSs usually operate in uncertain environments where various expected/unexpected events can affect their behaviors.<n>These events together with timing and synchronization incidents may result in various/different CPS behaviors and may cause a CPS to pass a test under some conditions and fail it under others.<n>When test cases are flaky, testing results are unreliable. To achieve more reliable test results, engineers may attempt to recognize flaky test cases and remove them.
• Score: 0.5161531917413708
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Cyber-Physical Systems (CPSs) have many applications, ranging from simple thermostat systems to autonomous driving systems and medical devices. Like all systems, they need to function correctly. To this end, validation techniques such as testing that can effectively reveal faults are required. Also, CPSs usually operate in uncertain environments where various expected/unexpected events can affect their behaviors. These events together with timing and synchronization incidents may result in various/different CPS behaviors and may cause a CPS to pass a test under some conditions and fail it under others. Such test cases are called ``flaky'' test cases and we call the conditions that are responsible for them ``flaky conditions'' and call these behaviors ``flaky behaviors''. When test cases are flaky, testing results are unreliable. To achieve more reliable test results, engineers may attempt to recognize flaky test cases and remove them. In this work, beginning with a test case generation and execution technique called HyTest that we created previously, we integrate TReVa, a new technique that validates testing results provided by HyTest and FlaRe, a new technique that recognizes flaky test cases and flaky conditions using hybrid models during the early stages of CPS development in just one additional round of testing. We present the results of an empirical study evaluating the effectiveness of our new approach (which we call HyTestTF). Our results show that correctly differentiate flaky test cases from non-flaky test cases

Related papers

• Delta Debugging for Cyber-Physical Systems with Flaky Test Executions [11.706481833926818]
  Simulation-based testing is widely used to validate Cyber-Physical Systems (CPSs)<n>Modern CPS simulators frequently exhibit non-deterministic (flaky) behavior, making failures difficult to reproduce and debug.<n>This paper presents three algorithms that combine statistical failure analysis, repeated executions, and environment-aware reduction to isolate minimal failure-inducing test inputs.
  arXiv  Detail & Related papers  (2026-07-28T13:14:39Z)
• Automated Test Oracles for Flaky Cyber-Physical System Simulators: Approach and Evaluation [1.5821080783312833]
  Simulation-based testing of cyber-physical systems (CPS) is costly due to the time-consuming execution of CPS simulators.<n>CPS simulators may be flaky, leading to inconsistent test outcomes and requiring repeated test re-execution for reliable test verdicts.<n>We propose assertion-based test oracles for CPS as sets of logical and arithmetic predicates defined over the inputs of the system under test.
  arXiv  Detail & Related papers  (2025-08-28T15:33:42Z)
• Do Test and Environmental Complexity Increase Flakiness? An Empirical Study of SAP HANA [47.29324864511411]
  Flaky tests fail seemingly at random without changes to the code. We study characteristics of tests and the test environment that potentially impact test flakiness.
  arXiv  Detail & Related papers  (2024-09-16T07:52:09Z)
• Context-Aware Fuzzing for Robustness Enhancement of Deep Learning Models [3.4790378315775348]
  We propose a novel testing metric called Contextual Confidence (CC) CC measures a test case through the surrounding samples of a test case in terms of their mean probability predicted to the prediction label of the test case. Clover computes the piece of difference between each such pair of a seed and a test case.
  arXiv  Detail & Related papers  (2024-07-17T09:31:10Z)
• GPT-HateCheck: Can LLMs Write Better Functional Tests for Hate Speech Detection? [50.53312866647302]
  HateCheck is a suite for testing fine-grained model functionalities on synthesized data. We propose GPT-HateCheck, a framework to generate more diverse and realistic functional tests from scratch. Crowd-sourced annotation demonstrates that the generated test cases are of high quality.
  arXiv  Detail & Related papers  (2024-02-23T10:02:01Z)
• Taming Timeout Flakiness: An Empirical Study of SAP HANA [47.29324864511411]
  Flaky tests negatively affect regression testing because they result in test failures that are not necessarily caused by code changes. Test timeouts are one contributing factor to such flaky test failures. Test flakiness rate ranges from 49% to 70%, depending on the number of repeated test executions.
  arXiv  Detail & Related papers  (2024-02-07T20:01:41Z)
• Test Case Generation and Test Oracle Support for Testing CPSs using Hybrid Models [2.6166087473624313]
  Cyber-Physical Systems (CPSs) play a central role in the behavior of a wide range of autonomous physical systems. CPSs are often specified iteratively as a sequence of models at different levels that can be tested via simulation systems. One such model is a hybrid automaton; these are used frequently for CPS applications and have the advantage of encapsulating both continuous and discrete CPS behaviors.
  arXiv  Detail & Related papers  (2023-09-14T19:08:09Z)
• Sequential Kernelized Independence Testing [77.237958592189]
  We design sequential kernelized independence tests inspired by kernelized dependence measures.<n>We demonstrate the power of our approaches on both simulated and real data.
  arXiv  Detail & Related papers  (2022-12-14T18:08:42Z)
• Efficient Test-Time Model Adaptation without Forgetting [60.36499845014649]
  Test-time adaptation seeks to tackle potential distribution shifts between training and testing data. We propose an active sample selection criterion to identify reliable and non-redundant samples. We also introduce a Fisher regularizer to constrain important model parameters from drastic changes.
  arXiv  Detail & Related papers  (2022-04-06T06:39:40Z)
• Noisy Adaptive Group Testing using Bayesian Sequential Experimental Design [63.48989885374238]
  When the infection prevalence of a disease is low, Dorfman showed 80 years ago that testing groups of people can prove more efficient than testing people individually. Our goal in this paper is to propose new group testing algorithms that can operate in a noisy setting.
  arXiv  Detail & Related papers  (2020-04-26T23:41:33Z)
• Dynamic Causal Effects Evaluation in A/B Testing with a Reinforcement Learning Framework [68.96770035057716]
  A/B testing is a business strategy to compare a new product with an old one in pharmaceutical, technological, and traditional industries. This paper introduces a reinforcement learning framework for carrying A/B testing in online experiments.
  arXiv  Detail & Related papers  (2020-02-05T10:25:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.