Correct Tests Are Not Enough: Measuring and Training Oracle Conversion in Specification-Based Test Generation

• URL: http://arxiv.org/abs/2609.05879v1
• Date: Sat, 05 Sep 2026 04:56:31 GMT
• Title: Correct Tests Are Not Enough: Measuring and Training Oracle Conversion in Specification-Based Test Generation
• Abstract summary: Generating tests from a natural-language specification requires both an input that exposes faulty behavior and a correct expected output.<n>We study this interaction through executable reward decomposition and suite-level oracle-conversion measurement.<n>Our generator jointly emits five input--output tests in one response.
• Score: 14.44743150931186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating tests from a natural-language specification requires both an input that exposes faulty behavior and a correct expected output. These requirements need not improve together: a model can increase test correctness by choosing easier inputs, or discover useful inputs whose expected outputs it cannot predict. We study this interaction through executable reward decomposition and suite-level oracle-conversion measurement. Our generator jointly emits five input--output tests in one response. During training, audited reference programs provide correctness feedback, while a fixed bank of faulty programs provides two utility signals: potential input kill and effective kill after checking the generated output. An additive GRPO objective preserves both signals without requiring execution at inference time. On an audited TC-Bench split with 506 training and 142 evaluation tasks, three independently trained Qwen3.5-9B runs at step 75 increase full-test correctness from 28.59\% to 42.54\%, input kill from 24.06\% to 25.27\%, and effective full kill from 12.23\% to 14.15\%. Matched 50-step ablations reveal a trade-off: removing kill rewards yields higher correctness and slightly higher full kill, but lowers input kill to 21.60\%. A fixed-input source--oracle crossover on 64 training-pool tasks attributes the principal NoKill-to-FullKill difference to harder input selection rather than worse output prediction on identical inputs. These results identify oracle conversion as a measurable bottleneck and show the benefits and limits of preserving input-utility feedback in joint test generation.

Related papers

• ExecCritic: Learn to Test, Test to Improve for Coding Agents [68.42713285082912]
  Execution feedback can guide coding agents toward correct repository repairs, but only when the tests capture the behavior requested by the issue.<n>We introduce ExecCritic, combining a test--verify--revise scaffold with a role-specific reinforcement learning recipe for training agents within it.<n>The scaffold separates test construction from source-code repair: a Test agent independently generates repository-native tests, a fail-closed harness qualifies and freezes them, and a Repair agent revises source code from their execution feedback without changing the tests.
  arXiv  Detail & Related papers  (2026-09-08T17:53:37Z)
• Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem [14.44743150931186]
  invalid or underspecified inputs can create spurious fault detections and apparent evolutionary gains.<n>We audit this failure mode in feedback-driven test generation using 142 development tasks, 114 locked external tasks, and 138 held-out tasks.<n>We propose an audit-and-placebo protocol that separates verifier artifacts, interaction scaffolding, and grounded feedback credit in evaluations of self-evolving test generators.
  arXiv  Detail & Related papers  (2026-08-20T04:29:35Z)
• Accuracy Hides How Language Models Fail: Measuring Failure States Under Matched Output Budgets [0.0]
  Language-model benchmarks collapse two distinct measurement questions into a single accuracy score.<n>We introduce a two-layer evaluation framework that separates scorer-independent execution evidence.
  arXiv  Detail & Related papers  (2026-07-27T11:04:17Z)
• Specification Grounding Drives Test Effectiveness for LLM Code [1.8620637029128544]
  Grounding the tests in the spec produces correct code +38 percentage points more often than this baseline.<n>It catches more real bugs wrongly far less correct code, cutting the false sensitivity rate from 33% (68% against Python standard-library oracle) to 0%.
  arXiv  Detail & Related papers  (2026-07-07T14:11:00Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Scaling Agentic Verifier for Competitive Coding [66.11758166379092]
  Large language models (LLMs) have demonstrated strong coding capabilities but still struggle to solve competitive programming problems correctly in a single attempt.<n>Execution-based re-ranking offers a promising test-time scaling strategy, yet existing methods are constrained by either difficult test case generation or inefficient random input sampling.<n>We propose Agentic Verifier, an execution-based agent that actively reasons about program behaviors and searches for highly discriminative test inputs.
  arXiv  Detail & Related papers  (2026-02-04T06:30:40Z)
• Nexus: Execution-Grounded Multi-Agent Test Oracle Synthesis [57.40527331817245]
  Test oracle generation in non-regression testing is a longstanding challenge in software engineering.<n>We introduce Nexus, a novel multi-agent framework to address this challenge.
  arXiv  Detail & Related papers  (2025-10-30T12:20:25Z)
• Directed Grammar-Based Test Generation [2.0948216657769616]
  This work proposes an automated test generation approach (called FdLoop)<n>FdLoop iteratively learns relevant input properties from existing inputs to drive the generation of goal-specific inputs.<n>We evaluate FdLoop using three (3) well-known input formats (JSON, CSS and JavaScript) and 20 open-source software.
  arXiv  Detail & Related papers  (2025-08-02T19:43:15Z)
• Learning to Generate Unit Tests for Automated Debugging [52.63217175637201]
  Unit tests (UTs) play an instrumental role in assessing code correctness as well as providing feedback to large language models (LLMs)<n>We propose UTGen, which teaches LLMs to generate unit test inputs that reveal errors along with their correct expected outputs.<n>We show that UTGen outperforms other LLM-based baselines by 7.59% based on a metric measuring the presence of both error-revealing UT inputs and correct UT outputs.
  arXiv  Detail & Related papers  (2025-02-03T18:51:43Z)
• Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference [72.61732440246954]
  Large pre-trained language models often lack logical consistency across test inputs. We propose a framework, ConCoRD, for boosting the consistency and accuracy of pre-trained NLP models. We show that ConCoRD consistently boosts accuracy and consistency of off-the-shelf closed-book QA and VQA models.
  arXiv  Detail & Related papers  (2022-11-21T21:58:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.