MTGP: Combining Metamorphic Testing and Genetic Programming

• URL: http://arxiv.org/abs/2301.08665v1
• Date: Fri, 20 Jan 2023 16:35:22 GMT
• Title: MTGP: Combining Metamorphic Testing and Genetic Programming
• Authors: Dominik Sobania, Martin Briesch, Philipp R\"ochner, Franz Rothlauf
• Abstract summary: Genetic programming is not yet mature enough for a practical use in real-world software development. We suggest MTGP, which combines metamorphic testing and genetic programming. We find that using metamorphic testing combined with labeled training cases leads to a higher generalization rate.
• Score: 1.6058099298620425
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Genetic programming is an evolutionary approach known for its performance in program synthesis. However, it is not yet mature enough for a practical use in real-world software development, since usually many training cases are required to generate programs that generalize to unseen test cases. As in practice, the training cases have to be expensively hand-labeled by the user, we need an approach to check the program behavior with a lower number of training cases. Metamorphic testing needs no labeled input/output examples. Instead, the program is executed multiple times, first on a given (randomly generated) input, followed by related inputs to check whether certain user-defined relations between the observed outputs hold. In this work, we suggest MTGP, which combines metamorphic testing and genetic programming and study its performance and the generalizability of the generated programs. Further, we analyze how the generalizability depends on the number of given labeled training cases. We find that using metamorphic testing combined with labeled training cases leads to a higher generalization rate than the use of labeled training cases alone in almost all studied configurations. Consequently, we recommend researchers to use metamorphic testing in their systems if the labeling of the training data is expensive.

Related papers

• CoCoEvo: Co-Evolution of Programs and Test Cases to Enhance Code Generation [3.113758966879047]
  CoCoEvo is a novel framework that simultaneously evolves programs and test cases. We show that CoCoEvo surpasses existing methods, achieving state-of-the-art performance in automated code generation and testing.
  arXiv  Detail & Related papers  (2025-02-15T13:52:30Z)
• Learning Semantics-aware Search Operators for Genetic Programming [0.20718016474717196]
  Fitness landscapes in test-based program synthesis are known to be extremely rugged. We propose a semantics-aware search operator that steers the search towards candidate programs that are valuable.
  arXiv  Detail & Related papers  (2025-02-06T23:46:04Z)
• Generational Computation Reduction in Informal Counterexample-Driven Genetic Programming [3.3748750222488657]
  Counterexample-driven genetic programming (CDGP) uses specifications provided as formal constraints to generate the training cases used to evaluate evolving programs. We show how the ideas underlying CDGP can also be applied using only user-provided training data, without formal specifications.
  arXiv  Detail & Related papers  (2024-08-08T01:06:28Z)
• Bounding Random Test Set Size with Computational Learning Theory [0.2999888908665658]
  We show how probabilistic approaches to answer this question in Machine Learning can be applied in our testing context. We are the first to enable this from only knowing the number of coverage targets in the source code. We validate this bound on a large set of Java units, and an autonomous driving system.
  arXiv  Detail & Related papers  (2024-05-27T10:15:16Z)
• Code Representation Pre-training with Complements from Program Executions [29.148208436656216]
  We propose FuzzPretrain to explore the dynamic information of programs revealed by their test cases and embed it into the feature representations of code as complements. FuzzyPretrain yielded more than 6%/9% mAP improvements on code search over its counterparts trained with only source code or AST.
  arXiv  Detail & Related papers  (2023-09-04T01:57:22Z)
• AdaNPC: Exploring Non-Parametric Classifier for Test-Time Adaptation [64.9230895853942]
  Domain generalization can be arbitrarily hard without exploiting target domain information. Test-time adaptive (TTA) methods are proposed to address this issue. In this work, we adopt Non-Parametric to perform the test-time Adaptation (AdaNPC)
  arXiv  Detail & Related papers  (2023-04-25T04:23:13Z)
• CodeT: Code Generation with Generated Tests [49.622590050797236]
  We explore the use of pre-trained language models to automatically generate test cases. CodeT executes the code solutions using the generated test cases, and then chooses the best solution. We evaluate CodeT on five different pre-trained models with both HumanEval and MBPP benchmarks.
  arXiv  Detail & Related papers  (2022-07-21T10:18:37Z)
• Fault-Aware Neural Code Rankers [64.41888054066861]
  We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
  arXiv  Detail & Related papers  (2022-06-04T22:01:05Z)
• Learning from Self-Sampled Correct and Partially-Correct Programs [96.66452896657991]
  We propose to let the model perform sampling during training and learn from both self-sampled fully-correct programs and partially-correct programs. We show that our use of self-sampled correct and partially-correct programs can benefit learning and help guide the sampling process. Our proposed method improves the pass@k performance by 3.1% to 12.3% compared to learning from a single reference program with MLE.
  arXiv  Detail & Related papers  (2022-05-28T03:31:07Z)
• Enforcing Consistency in Weakly Supervised Semantic Parsing [68.2211621631765]
  We explore the use of consistency between the output programs for related inputs to reduce the impact of spurious programs. We find that a more consistent formalism leads to improved model performance even without consistency-based training.
  arXiv  Detail & Related papers  (2021-07-13T03:48:04Z)
• Soft Genetic Programming Binary Classifiers [0.0]
  "Soft" genetic programming (SGP) has been developed, which allows the logical operator tree to be more flexible and find dependencies in datasets. This article discusses a method for constructing binary classifiers using the SGP technique.
  arXiv  Detail & Related papers  (2021-01-21T17:43:11Z)
• Unshuffling Data for Improved Generalization [65.57124325257409]
  Generalization beyond the training distribution is a core challenge in machine learning. We show that partitioning the data into well-chosen, non-i.i.d. subsets treated as multiple training environments can guide the learning of models with better out-of-distribution generalization.
  arXiv  Detail & Related papers  (2020-02-27T03:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.