Metamorphic Relation Prioritization for Effective Regression Testing

• URL: http://arxiv.org/abs/2109.09798v1
• Date: Mon, 20 Sep 2021 19:06:17 GMT
• Title: Metamorphic Relation Prioritization for Effective Regression Testing
• Authors: Madhusudan Srinivasan and Upulee Kanewala
• Abstract summary: We propose approaches to prioritize metamorphic relations (MRs) to improve the efficiency and effectiveness of regression testing. We present two MR prioritization approaches: (1) fault-based and (2) coverage-based. Our results show that fault-based MR prioritization leads to reducing the number of source and follow-up test cases that needs to be executed.
• Score: 11.033596835816422
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Metamorphic testing (MT) is widely used for testing programs that face the oracle problem. It uses a set of metamorphic relations (MRs), which are relations among multiple inputs and their corresponding outputs to determine whether the program under test is faulty. Typically, MRs vary in their ability to detect faults in the program under test, and some MRs tend to detect the same set of faults. In this paper, we propose approaches to prioritize MRs to improve the efficiency and effectiveness of MT for regression testing. We present two MR prioritization approaches: (1) fault-based and (2) coverage-based. To evaluate these MR prioritization approaches, we conduct experiments on three complex open-source software systems. Our results show that the MR prioritization approaches developed by us significantly outperform the current practice of executing the source and follow-up test cases of the MRs in an ad-hoc manner in terms of fault detection effectiveness. Further, fault-based MR prioritization leads to reducing the number of source and follow-up test cases that needs to be executed as well as reducing the average time taken to detect a fault, which would result in saving time and cost during the testing process.

Related papers

• Optimizing Metamorphic Testing: Prioritizing Relations Through Execution Profile Dissimilarity [2.6749261270690434]
  An oracle determines whether the output of a program for executed test cases is correct. For machine learning programs, such an oracle is often unavailable or impractical to apply. Prioritizing MRs enhances fault detection effectiveness and improves testing efficiency.
  arXiv  Detail & Related papers  (2024-11-14T04:14:30Z)
• An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking [50.81324768683995]
  FIRST is a novel approach that integrates a learning-to-rank objective and leveraging the logits of only the first generated token. We extend the evaluation of FIRST to the TREC Deep Learning datasets (DL19-22), validating its robustness across diverse domains. Our experiments confirm that fast reranking with single-token logits does not compromise out-of-domain reranking quality.
  arXiv  Detail & Related papers  (2024-11-08T12:08:17Z)
• Segment-Based Test Case Prioritization: A Multi-objective Approach [8.972346309150199]
  Test case prioritization ( TCP) is a cost-efficient solution to schedule test cases in an execution order that maximizes an objective function. We introduce a multi-objective optimization approach to prioritize UI test cases using evolutionary search algorithms and four coverage criteria. Our approach significantly outperforms other methods in terms of Average Percentage of Faults Detected (APFD) and APFD with Cost.
  arXiv  Detail & Related papers  (2024-08-01T16:51:01Z)
• Improving Bias Correction Standards by Quantifying its Effects on Treatment Outcomes [54.18828236350544]
  Propensity score matching (PSM) addresses selection biases by selecting comparable populations for analysis. Different matching methods can produce significantly different Average Treatment Effects (ATE) for the same task, even when meeting all validation criteria. To address this issue, we introduce a novel metric, A2A, to reduce the number of valid matches.
  arXiv  Detail & Related papers  (2024-07-20T12:42:24Z)
• MR-Ben: A Meta-Reasoning Benchmark for Evaluating System-2 Thinking in LLMs [55.20845457594977]
  Large language models (LLMs) have shown increasing capability in problem-solving and decision-making. We present a process-based benchmark MR-Ben that demands a meta-reasoning skill. Our meta-reasoning paradigm is especially suited for system-2 slow thinking.
  arXiv  Detail & Related papers  (2024-06-20T03:50:23Z)
• Fuzzy Inference System for Test Case Prioritization in Software Testing [0.0]
  Test case prioritization ( TCP) is a vital strategy to enhance testing efficiency. This paper introduces a novel fuzzy logic-based approach to automate TCP.
  arXiv  Detail & Related papers  (2024-04-25T08:08:54Z)
• Towards a Complete Metamorphic Testing Pipeline [56.75969180129005]
  Metamorphic Testing (MT) addresses the test oracle problem by examining the relationships between input-output pairs in consecutive executions of the System Under Test (SUT) These relations, known as Metamorphic Relations (MRs), specify the expected output changes resulting from specific input changes. Our research aims to develop methods and tools that assist testers in generating MRs, defining constraints, and providing explainability for MR outcomes.
  arXiv  Detail & Related papers  (2023-09-30T10:49:22Z)
• On Pitfalls of Test-Time Adaptation [82.8392232222119]
  Test-Time Adaptation (TTA) has emerged as a promising approach for tackling the robustness challenge under distribution shifts. We present TTAB, a test-time adaptation benchmark that encompasses ten state-of-the-art algorithms, a diverse array of distribution shifts, and two evaluation protocols.
  arXiv  Detail & Related papers  (2023-06-06T09:35:29Z)
• MR-Scout: Automated Synthesis of Metamorphic Relations from Existing Test Cases [9.00297842984345]
  We propose MR-Scout to automatically synthesize MRs from test cases in open-source software projects. Over 97% of codified MRs are of high quality for automated test case generation. Our qualitative study shows that 55.76% to 76.92% of codified MRs are easily comprehensible for developers.
  arXiv  Detail & Related papers  (2023-04-15T12:53:32Z)
• Improving a State-of-the-Art Heuristic for the Minimum Latency Problem with Data Mining [69.00394670035747]
  Hybrid metaheuristics have become a trend in operations research. A successful example combines the Greedy Randomized Adaptive Search Procedures (GRASP) and data mining techniques.
  arXiv  Detail & Related papers  (2019-08-28T13:12:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.