An approach for performance requirements verification and test environments generation

• URL: http://arxiv.org/abs/2403.00099v1
• Date: Thu, 29 Feb 2024 19:59:26 GMT
• Title: An approach for performance requirements verification and test environments generation
• Authors: Waleed Abdeen, Xingru Chen, Michael Unterkalmsteiner
• Abstract summary: We conducted a systematic mapping study on model-based performance testing. We studied natural language software requirements specifications in order to understand which and how performance requirements are typically specified. Since none of the identified MBT techniques supported a major benefit of modeling, we developed the Performance Requirements verificatiOn and Test Environments generaTion approach.
• Score: 1.359087929215203
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Model-based testing (MBT) is a method that supports the design and execution of test cases by models that specify the intended behaviors of a system under test. While systematic literature reviews on MBT in general exist, the state of the art on modeling and testing performance requirements has seen much less attention. Therefore, we conducted a systematic mapping study on model-based performance testing. Then, we studied natural language software requirements specifications in order to understand which and how performance requirements are typically specified. Since none of the identified MBT techniques supported a major benefit of modeling, namely identifying faults in requirements specifications, we developed the Performance Requirements verificatiOn and Test EnvironmentS generaTion approach (PRO-TEST). Finally, we evaluated PRO-TEST on 149 requirements specifications. We found and analyzed 57 primary studies from the systematic mapping study and extracted 50 performance requirements models. However, those models don't achieve the goals of MBT, which are validating requirements, ensuring their testability, and generating the minimum required test cases. We analyzed 77 Software Requirements Specification (SRS) documents, extracted 149 performance requirements from those SRS, and illustrate that with PRO-TEST we can model performance requirements, find issues in those requirements and detect missing ones. We detected three not-quantifiable requirements, 43 not-quantified requirements, and 180 underspecified parameters in the 149 modeled performance requirements. Furthermore, we generated 96 test environments from those models. By modeling performance requirements with PRO-TEST, we can identify issues in the requirements related to their ambiguity, measurability, and completeness. Additionally, it allows to generate parameters for test environments.

Related papers

• CLOVER: A Test Case Generation Benchmark with Coverage, Long-Context, and Verification [71.34070740261072]
  This paper presents a benchmark, CLOVER, to evaluate models' capabilities in generating and completing test cases. The benchmark is containerized for code execution across tasks, and we will release the code, data, and construction methodologies.
  arXiv  Detail & Related papers  (2025-02-12T21:42:56Z)
• Formal Analysis of the Contract Automata Runtime Environment with Uppaal: Modelling, Verification and Testing [0.3807314298073301]
  A distributed runtime application called tt CARE has been introduced to realise service applications specified using a dialect of finite-state automata. We detail the formal modelling, verification and testing of tt CARE.
  arXiv  Detail & Related papers  (2025-01-22T15:03:25Z)
• System Test Case Design from Requirements Specifications: Insights and Challenges of Using ChatGPT [1.9282110216621835]
  This paper explores the effectiveness of using Large Language Models (LLMs) to generate test case designs from Software Requirements Specification (SRS) documents. About 87 percent of the generated test cases were valid, with the remaining 13 percent either not applicable or redundant.
  arXiv  Detail & Related papers  (2024-12-04T20:12:27Z)
• Benchmarks as Microscopes: A Call for Model Metrology [76.64402390208576]
  Modern language models (LMs) pose a new challenge in capability assessment. To be confident in our metrics, we need a new discipline of model metrology.
  arXiv  Detail & Related papers  (2024-07-22T17:52:12Z)
• Using Quality Attribute Scenarios for ML Model Test Case Generation [3.9111051646728527]
  Current practice for machine learning (ML) model testing prioritizes testing for model performance. This paper presents an approach based on quality attribute (QA) scenarios to elicit and define system- and model-relevant test cases. The QA-based approach has been integrated into MLTE, a process and tool to support ML model test and evaluation.
  arXiv  Detail & Related papers  (2024-06-12T18:26:42Z)
• Coupled Requirements-driven Testing of CPS: From Simulation To Reality [5.7736484832934325]
  Failures in safety-critical Cyber-Physical Systems (CPS) can lead to severe incidents impacting physical infrastructure or even harming humans. Current simulation and field testing practices, particularly in the domain of small Unmanned Aerial Systems (sUAS), are ad-hoc and lack a thorough, structured testing process. We have developed an initial framework for validating CPS, specifically focusing on sUAS and robotic applications.
  arXiv  Detail & Related papers  (2024-03-24T20:32:12Z)
• Can You Rely on Your Model Evaluation? Improving Model Evaluation with Synthetic Test Data [75.20035991513564]
  We introduce 3S Testing, a deep generative modeling framework to facilitate model evaluation. Our experiments demonstrate that 3S Testing outperforms traditional baselines. These results raise the question of whether we need a paradigm shift away from limited real test data towards synthetic test data.
  arXiv  Detail & Related papers  (2023-10-25T10:18:44Z)
• A Requirements-Driven Platform for Validating Field Operations of Small Uncrewed Aerial Vehicles [48.67061953896227]
  DroneReqValidator (DRV) allows sUAS developers to define the operating context, configure multi-sUAS mission requirements, specify safety properties, and deploy their own custom sUAS applications in a high-fidelity 3D environment. The DRV Monitoring system collects runtime data from sUAS and the environment, analyzes compliance with safety properties, and captures violations.
  arXiv  Detail & Related papers  (2023-07-01T02:03:49Z)
• From Static Benchmarks to Adaptive Testing: Psychometrics in AI Evaluation [60.14902811624433]
  We discuss a paradigm shift from static evaluation methods to adaptive testing. This involves estimating the characteristics and value of each test item in the benchmark and dynamically adjusting items in real-time. We analyze the current approaches, advantages, and underlying reasons for adopting psychometrics in AI evaluation.
  arXiv  Detail & Related papers  (2023-06-18T09:54:33Z)
• Model-based Analysis and Specification of Functional Requirements and Tests for Complex Automotive Systems [0.19837121116620585]
  We propose a technique that starts with the early identification of validation concerns from a stakeholder perspective. We develop a Model-Based Systems Engineering (MBSE) methodology to ensure complete and consistent requirements and test specifications. Our study corroborates that our methodology is applicable and improves existing requirements and test specification processes.
  arXiv  Detail & Related papers  (2022-09-03T18:24:32Z)
• On the Limits of Evaluating Embodied Agent Model Generalization Using Validation Sets [101.28658250723804]
  This paper experiments with augmenting a transformer model with modules that effectively utilize a wider field of view and learn to choose whether the next step requires a navigation or manipulation action. We observe that the proposed modules resulted in improved, and in fact state-of-the-art performance on an unseen validation set of a popular benchmark dataset, ALFRED. We highlight this result as we believe it may be a wider phenomenon in machine learning tasks but primarily noticeable only in benchmarks that limit evaluations on test splits.
  arXiv  Detail & Related papers  (2022-05-18T23:52:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.