MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases

• URL: http://arxiv.org/abs/2304.07548v4
• Date: Tue, 25 Jun 2024 02:42:13 GMT
• Title: MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases
• Authors: Congying Xu, Valerio Terragni, Hengcheng Zhu, Jiarong Wu, Shing-Chi Cheung,
• Abstract summary: We propose MR-Scout to automatically synthesize MRs from test cases in open-source software projects. Over 97% of codified MRs are of high quality for automated test case generation. Our qualitative study shows that 55.76% to 76.92% of codified MRs are easily comprehensible for developers.
• Score: 9.00297842984345
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Metamorphic Testing (MT) alleviates the oracle problem by defining oracles based on metamorphic relations (MRs), that govern multiple related inputs and their outputs. However, designing MRs is challenging, as it requires domain-specific knowledge. This hinders the widespread adoption of MT. We observe that developer-written test cases can embed domain knowledge that encodes MRs. Such encoded MRs could be synthesized for testing not only their original programs but also other programs that share similar functionalities. In this paper, we propose MR-Scout to automatically synthesize MRs from test cases in open-source software (OSS) projects. MR-Scout first discovers MR-encoded test cases (MTCs), and then synthesizes the encoded MRs into parameterized methods (called codified MRs), and filters out MRs that demonstrate poor quality for new test case generation. MR-Scout discovered over 11,000 MTCs from 701 OSS projects. Experimental results show that over 97% of codified MRs are of high quality for automated test case generation, demonstrating the practical applicability of MR-Scout. Furthermore, codified-MRs-based tests effectively enhance the test adequacy of programs with developer-written tests, leading to 13.52% and 9.42% increases in line coverage and mutation score, respectively. Our qualitative study shows that 55.76% to 76.92% of codified MRs are easily comprehensible for developers.

Related papers

• Metamorphic Testing for Smart Contract Validation:A Case Study of Ethereum-Based Crowdfunding Contracts [0.11470070927586014]
  A key challenge in the testing of smart contracts is the oracle problem, where the exact expected outcomes are not well defined. We use Metamorphic Relations (MRs) to validate smart contracts. MRs define how output should change relative to specific input, determining whether the tests pass or fail. Our results show that our MRs detected 25.65% of the total mutants generated, with the most effective MRs achieving a mutant-killing rate of 89%.
  arXiv  Detail & Related papers  (2025-01-17T05:04:23Z)
• GenMorph: Automatically Generating Metamorphic Relations via Genetic Programming [14.417892080404544]
  GenMorph is a technique to automatically generate Metamorphic Relations (MRs) for Java methods. It uses an evolutionary algorithm to search for effective test oracles that trigger no false alarms and expose software faults in the method under test. Our results show that GenMorph generates effective MRs for 18 out of 23 methods (mutation score >20%), and it can increase Randoop's fault detection capability in 7 out of 23 methods.
  arXiv  Detail & Related papers  (2023-12-23T17:20:45Z)
• Towards a Complete Metamorphic Testing Pipeline [56.75969180129005]
  Metamorphic Testing (MT) addresses the test oracle problem by examining the relationships between input-output pairs in consecutive executions of the System Under Test (SUT) These relations, known as Metamorphic Relations (MRs), specify the expected output changes resulting from specific input changes. Our research aims to develop methods and tools that assist testers in generating MRs, defining constraints, and providing explainability for MR outcomes.
  arXiv  Detail & Related papers  (2023-09-30T10:49:22Z)
• Exploring a Test Data-Driven Method for Selecting and Constraining Metamorphic Relations [46.889513596156185]
  This paper presents a preliminary evaluation of MetaTrimmer, a method for selecting and constraining Metamorphic Relations based on test data. The novelty of MetaTrimmer is its avoidance of complex prediction models that require labeled datasets regarding the applicability of MRs. In a preliminary evaluation, MetaTrimmer shows the potential to overcome existing limitations and enhance MR effectiveness.
  arXiv  Detail & Related papers  (2023-07-28T12:27:34Z)
• Bug or not Bug? Analysing the Reasons Behind Metamorphic Relation Violations [46.889513596156185]
  Metamorphic Testing (MT) is a testing technique that can effectively alleviate the oracle problem. MT uses Metamorphic Relations (MRs) to determine if a test case passes or fails. We develop a method for refining MRs to offer hints as to whether a violation results from a bug or arises from the MR not being matched to certain test data.
  arXiv  Detail & Related papers  (2023-05-16T17:42:37Z)
• Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models [57.80514758695275]
  Using large language models (LLMs) for assessing the quality of machine translation (MT) achieves state-of-the-art performance at the system level. We propose a new prompting method called textbftextttError Analysis Prompting (EAPrompt) This technique emulates the commonly accepted human evaluation framework - Multidimensional Quality Metrics (MQM) and textitproduces explainable and reliable MT evaluations at both the system and segment level.
  arXiv  Detail & Related papers  (2023-03-24T05:05:03Z)
• Advancing Radiograph Representation Learning with Masked Record Modeling [52.04899592688968]
  We formulate the self- and report-completion as two complementary objectives and present a unified framework based on masked record modeling (MRM) MRM reconstructs masked image patches and masked report tokens following a multi-task scheme to learn knowledge-enhanced semantic representations. Specifically, we find that MRM offers superior performance in label-efficient fine-tuning.
  arXiv  Detail & Related papers  (2023-01-30T18:33:32Z)
• Effective Mutation Rate Adaptation through Group Elite Selection [50.88204196504888]
  This paper introduces the Group Elite Selection of Mutation Rates (GESMR) algorithm. GESMR co-evolves a population of solutions and a population of MRs, such that each MR is assigned to a group of solutions. With the same number of function evaluations and with almost no overhead, GESMR converges faster and to better solutions than previous approaches.
  arXiv  Detail & Related papers  (2022-04-11T01:08:26Z)
• Metamorphic Relation Prioritization for Effective Regression Testing [11.033596835816422]
  We propose approaches to prioritize metamorphic relations (MRs) to improve the efficiency and effectiveness of regression testing. We present two MR prioritization approaches: (1) fault-based and (2) coverage-based. Our results show that fault-based MR prioritization leads to reducing the number of source and follow-up test cases that needs to be executed.
  arXiv  Detail & Related papers  (2021-09-20T19:06:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.