Bug or not Bug? Analysing the Reasons Behind Metamorphic Relation Violations

• URL: http://arxiv.org/abs/2305.09640v1
• Date: Tue, 16 May 2023 17:42:37 GMT
• Title: Bug or not Bug? Analysing the Reasons Behind Metamorphic Relation Violations
• Authors: Alejandra Duque-Torres, Dietmar Pfahl, Claus Klammer and Stefan Fischer
• Abstract summary: Metamorphic Testing (MT) is a testing technique that can effectively alleviate the oracle problem. MT uses Metamorphic Relations (MRs) to determine if a test case passes or fails. We develop a method for refining MRs to offer hints as to whether a violation results from a bug or arises from the MR not being matched to certain test data.
• Score: 46.889513596156185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Metamorphic Testing (MT) is a testing technique that can effectively alleviate the oracle problem. MT uses Metamorphic Relations (MRs) to determine if a test case passes or fails. MRs specify how the outputs should vary in response to specific input changes when executing the System Under Test (SUT). If a particular MR is violated for at least one test input (and its change), there is a high probability that the SUT has a fault. On the other hand, if a particular MR is not violated, it does not guarantee that the SUT is fault free. However, deciding if the MR is being violated due to a bug or because the MR does not hold/fit for particular conditions generated by specific inputs remains a manual task and unexplored. In this paper, we develop a method for refining MRs to offer hints as to whether a violation results from a bug or arises from the MR not being matched to certain test data under specific circumstances. In our initial proof-of-concept, we derive the relevant information from rules using the Association Rule Mining (ARM) technique. In our initial proof-of-concept, we validate our method on a toy example and discuss the lessons learned from our experiments. Our proof-of-concept demonstrates that our method is applicable and that we can provide suggestions that help strengthen the test suite for regression testing purposes.

Related papers

• Loophole-free test of macroscopic realism via high-order correlations of measurement [6.768813867436463]
  Test of macroscopic realism (MR) is key to understanding the foundation of quantum mechanics. We propose a general inequality based on high-order correlations of measurements for a loophole-free test of MR at the weak signal limit. We demonstrate that the inequality can be broken by a quantum spin model.
  arXiv  Detail & Related papers  (2024-01-10T16:19:59Z)
• GenMorph: Automatically Generating Metamorphic Relations via Genetic Programming [14.417892080404544]
  GenMorph is a technique to automatically generate Metamorphic Relations (MRs) for Java methods. It uses an evolutionary algorithm to search for effective test oracles that trigger no false alarms and expose software faults in the method under test. Our results show that GenMorph generates effective MRs for 18 out of 23 methods (mutation score >20%), and it can increase Randoop's fault detection capability in 7 out of 23 methods.
  arXiv  Detail & Related papers  (2023-12-23T17:20:45Z)
• R-Tuning: Instructing Large Language Models to Say `I Don't Know' [66.11375475253007]
  Large language models (LLMs) have revolutionized numerous domains with their impressive performance but still face their challenges. Previous instruction tuning methods force the model to complete a sentence no matter whether the model knows the knowledge or not. We present a new approach called Refusal-Aware Instruction Tuning (R-Tuning) Experimental results demonstrate R-Tuning effectively improves a model's ability to answer known questions and refrain from answering unknown questions.
  arXiv  Detail & Related papers  (2023-11-16T08:45:44Z)
• Towards a Complete Metamorphic Testing Pipeline [56.75969180129005]
  Metamorphic Testing (MT) addresses the test oracle problem by examining the relationships between input-output pairs in consecutive executions of the System Under Test (SUT) These relations, known as Metamorphic Relations (MRs), specify the expected output changes resulting from specific input changes. Our research aims to develop methods and tools that assist testers in generating MRs, defining constraints, and providing explainability for MR outcomes.
  arXiv  Detail & Related papers  (2023-09-30T10:49:22Z)
• Exploring a Test Data-Driven Method for Selecting and Constraining Metamorphic Relations [46.889513596156185]
  This paper presents a preliminary evaluation of MetaTrimmer, a method for selecting and constraining Metamorphic Relations based on test data. The novelty of MetaTrimmer is its avoidance of complex prediction models that require labeled datasets regarding the applicability of MRs. In a preliminary evaluation, MetaTrimmer shows the potential to overcome existing limitations and enhance MR effectiveness.
  arXiv  Detail & Related papers  (2023-07-28T12:27:34Z)
• MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases [9.00297842984345]
  We propose MR-Scout to automatically synthesize MRs from test cases in open-source software projects. Over 97% of codified MRs are of high quality for automated test case generation. Our qualitative study shows that 55.76% to 76.92% of codified MRs are easily comprehensible for developers.
  arXiv  Detail & Related papers  (2023-04-15T12:53:32Z)
• Sequential Kernelized Independence Testing [101.22966794822084]
  We design sequential kernelized independence tests inspired by kernelized dependence measures. We demonstrate the power of our approaches on both simulated and real data.
  arXiv  Detail & Related papers  (2022-12-14T18:08:42Z)
• Using Metamorphic Relations to Verify and Enhance Artcode Classification [39.36253474867746]
  An example of an area facing the oracle problem is automatic image classification, using machine learning to classify an input image as one of a set of predefined classes. An approach to software testing that alleviates the oracle problem is metamorphic testing (MT) This paper examines the problem of classifying images containing visually hidden markers called Artcodes, and applies MT to verify and enhance the trained classifiers.
  arXiv  Detail & Related papers  (2021-08-05T15:54:56Z)
• Maximum Mean Discrepancy Test is Aware of Adversarial Attacks [122.51040127438324]
  The maximum mean discrepancy (MMD) test could in principle detect any distributional discrepancy between two datasets. It has been shown that the MMD test is unaware of adversarial attacks.
  arXiv  Detail & Related papers  (2020-10-22T03:42:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.