Retromorphic Testing: A New Approach to the Test Oracle Problem

• URL: http://arxiv.org/abs/2310.06433v1
• Date: Tue, 10 Oct 2023 09:03:01 GMT
• Title: Retromorphic Testing: A New Approach to the Test Oracle Problem
• Authors: Boxi Yu, Qiuyang Mang, Qingshuo Guo, Pinjia He
• Abstract summary: A test oracle serves as a criterion or mechanism to assess the correspondence between software output and the anticipated behavior for a given input set. Inspired by the mathematical concept of inverse function, we present Retromorphic Testing, a novel black-box testing methodology. This paper presents its three testing modes with illustrative use cases across diverse programs, including algorithms, traditional software, and AI applications.
• Score: 4.183247207161993
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A test oracle serves as a criterion or mechanism to assess the correspondence between software output and the anticipated behavior for a given input set. In automated testing, black-box techniques, known for their non-intrusive nature in test oracle construction, are widely used, including notable methodologies like differential testing and metamorphic testing. Inspired by the mathematical concept of inverse function, we present Retromorphic Testing, a novel black-box testing methodology. It leverages an auxiliary program in conjunction with the program under test, which establishes a dual-program structure consisting of a forward program and a backward program. The input data is first processed by the forward program and then its program output is reversed to its original input format using the backward program. In particular, the auxiliary program can operate as either the forward or backward program, leading to different testing modes. The process concludes by examining the relationship between the initial input and the transformed output within the input domain. For example, to test the implementation of the sine function $\sin(x)$, we can employ its inverse function, $\arcsin(x)$, and validate the equation $x = \sin(\arcsin(x)+2k\pi), \forall k \in \mathbb{Z}$. In addition to the high-level concept of Retromorphic Testing, this paper presents its three testing modes with illustrative use cases across diverse programs, including algorithms, traditional software, and AI applications.

Related papers

• Combining Tests and Proofs for Better Software Verification [30.905701881162685]
  A different perspective is emerging, in which testing and proving are complementary rather than competing techniques for producing software of verified quality.<n>A counterexample is an input combination that makes the program fail.<n>One can, however, apply counterexample generation to incorrect programs, as a tool for automatic test generation.<n>We can also use these mechanisms to help produce program fixes for incorrect programs, with a guarantee that the fixes are correct.
  arXiv  Detail & Related papers  (2026-01-21T22:39:17Z)
• Program Synthesis via Test-Time Transduction [26.30808249424997]
  We introduce transductive program synthesis, a new formulation of the program synthesis task that explicitly leverages test inputs during synthesis.<n>We evaluate our approach on four benchmarks: Playgol, MBPP+, 1D-ARC, and programmatic world modeling on MiniGrid.<n>We demonstrate that our method significantly improves program synthesis in both accuracy and efficiency.
  arXiv  Detail & Related papers  (2025-09-22T06:53:32Z)
• Directed Grammar-Based Test Generation [2.0948216657769616]
  This work proposes an automated test generation approach (called FdLoop)<n>FdLoop iteratively learns relevant input properties from existing inputs to drive the generation of goal-specific inputs.<n>We evaluate FdLoop using three (3) well-known input formats (JSON, CSS and JavaScript) and 20 open-source software.
  arXiv  Detail & Related papers  (2025-08-02T19:43:15Z)
• Test3R: Learning to Reconstruct 3D at Test Time [58.0912500917036]
  Test3R is a surprisingly simple test-time learning technique that significantly boosts geometric accuracy.<n>Our technique significantly outperforms previous state-of-the-art methods on the 3D reconstruction and multi-view depth estimation tasks.
  arXiv  Detail & Related papers  (2025-06-16T17:56:22Z)
• Automated Test Generation from Program Documentation Encoded in Code Comments [4.696083734269232]
  This paper introduces a novel test generation technique that exploits the code-comment documentation constructively. We deliver test cases with names and oracles properly contextualized on the target behaviors.
  arXiv  Detail & Related papers  (2025-04-29T20:23:56Z)
• ReasonAgain: Using Extractable Symbolic Programs to Evaluate Mathematical Reasoning [54.70811660561151]
  Existing math datasets evaluate the reasoning abilities of large language models (LLMs) by either using the final answer or the intermediate reasoning steps derived from static examples. We seek to use symbolic programs as a means for automated evaluation if a model can consistently produce correct final answers across various inputs to the program. We observe significant accuracy drops using our proposed evaluation compared with original static examples, suggesting the fragility of math reasoning in state-of-the-art LLMs.
  arXiv  Detail & Related papers  (2024-10-24T18:02:37Z)
• Improved Tree Search for Automatic Program Synthesis [91.3755431537592]
  A key element is being able to perform an efficient search in the space of valid programs. Here, we suggest a variant of MCTS that leads to state of the art results on two vastly different DSLs.
  arXiv  Detail & Related papers  (2023-03-13T15:09:52Z)
• Genetic Micro-Programs for Automated Software Testing with Large Path Coverage [0.0]
  Existing software testing techniques focus on utilising search algorithms to discover input values that achieve high execution path coverage. This paper outlines a novel genetic programming framework, where the evolved solutions are not input values, but micro-programs that can repeatedly generate input values. We argue that our approach can be generalised such as to be applied to many different software systems, and is thus not specific to merely the particular software component on which it was trained.
  arXiv  Detail & Related papers  (2023-02-14T18:47:21Z)
• MTGP: Combining Metamorphic Testing and Genetic Programming [1.6058099298620425]
  Genetic programming is not yet mature enough for a practical use in real-world software development. We suggest MTGP, which combines metamorphic testing and genetic programming. We find that using metamorphic testing combined with labeled training cases leads to a higher generalization rate.
  arXiv  Detail & Related papers  (2023-01-20T16:35:22Z)
• NAPG: Non-Autoregressive Program Generation for Hybrid Tabular-Textual Question Answering [52.10214317661547]
  Current numerical reasoning methods autoregressively decode program sequences. The accuracy of program generation drops sharply as the decoding steps unfold due to error propagation. In this paper, we propose a non-autoregressive program generation framework.
  arXiv  Detail & Related papers  (2022-11-07T11:25:21Z)
• Using Quantum Computers to Speed Up Dynamic Testing of Software [0.0]
  As the number and possible values of input parameters increase, the cost of dynamic testing rises. This paper examines whether quantum computers (QCs) can help speed up the dynamic testing of programs written for classical computers.
  arXiv  Detail & Related papers  (2022-09-11T13:28:26Z)
• Fault-Aware Neural Code Rankers [64.41888054066861]
  We propose fault-aware neural code rankers that can predict the correctness of a sampled program without executing it. Our fault-aware rankers can significantly increase the pass@1 accuracy of various code generation models.
  arXiv  Detail & Related papers  (2022-06-04T22:01:05Z)
• Learning from Self-Sampled Correct and Partially-Correct Programs [96.66452896657991]
  We propose to let the model perform sampling during training and learn from both self-sampled fully-correct programs and partially-correct programs. We show that our use of self-sampled correct and partially-correct programs can benefit learning and help guide the sampling process. Our proposed method improves the pass@k performance by 3.1% to 12.3% compared to learning from a single reference program with MLE.
  arXiv  Detail & Related papers  (2022-05-28T03:31:07Z)
• Towards Neural Functional Program Evaluation [0.5586191108738562]
  We introduce a new program generation mechanism that allows control over syntactic sugar for semantically equivalent programs. Experiments reveal that neural functional program evaluation performs surprisingly well, achieving high 90% exact program match scores.
  arXiv  Detail & Related papers  (2021-12-09T00:20:29Z)
• IReEn: Reverse-Engineering of Black-Box Functions via Iterative Neural Program Synthesis [70.61283188380689]
  We investigate the problem of revealing the functionality of a black-box agent. We do not rely on privileged information on the black box, but rather investigate the problem under a weaker assumption of having only access to inputs and outputs of the program. Our results show that the proposed approach outperforms the state-of-the-art on this challenge by finding an approximately functional equivalent program in 78% of cases.
  arXiv  Detail & Related papers  (2020-06-18T17:50:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.