Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem

• URL: http://arxiv.org/abs/2608.19626v1
• Date: Thu, 20 Aug 2026 04:29:35 GMT
• Title: Auditing and Decomposing Feedback-Driven Evolution in LLM Test Generation under the Oracle Problem
• Abstract summary: invalid or underspecified inputs can create spurious fault detections and apparent evolutionary gains.<n>We audit this failure mode in feedback-driven test generation using 142 development tasks, 114 locked external tasks, and 138 held-out tasks.<n>We propose an audit-and-placebo protocol that separates verifier artifacts, interaction scaffolding, and grounded feedback credit in evaluations of self-evolving test generators.
• Score: 14.44743150931186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Execution feedback is often treated as a self-verifying signal for improving LLM-generated tests. However, when generated inputs are executed on a single accepted program and its outputs are used as ground truth, invalid or underspecified inputs can create spurious fault detections and apparent evolutionary gains. We audit this failure mode in feedback-driven test generation using 142 development tasks, 114 locked external tasks, and 138 held-out tasks, with two code models, three seeds, and fault-cross-fitted real submissions. On external inputs for which three accepted implementations agree, generated outputs match the panel on only 27.79% and 50.12% of cases. A single-reference oracle inflates the measured gain from evolution by 9.46-14.85 percentage points; after auditing, equal-budget independent resampling outperforms mutation-based evolution by 6.01-18.83 points. We further compare a genuine three-round feedback loop with a density-matched placebo. External Real-Placebo differences are +0.13 and -0.50 points, while held-out differences are +1.99 and +0.28 points and do not provide robust evidence of fine-grained feedback benefit. A blinded semantic audit by two software engineering doctoral students classifies 94.41% of panel-disconfirmed inputs as invalid but 3.60% as valid, showing that panel disagreement is informative but not semantic proof. We propose an audit-and-placebo protocol that separates verifier artifacts, interaction scaffolding, and grounded feedback credit in evaluations of self-evolving test generators.

Related papers

• Correct Tests Are Not Enough: Measuring and Training Oracle Conversion in Specification-Based Test Generation [14.44743150931186]
  Generating tests from a natural-language specification requires both an input that exposes faulty behavior and a correct expected output.<n>We study this interaction through executable reward decomposition and suite-level oracle-conversion measurement.<n>Our generator jointly emits five input--output tests in one response.
  arXiv  Detail & Related papers  (2026-09-05T04:56:31Z)
• Rethinking Self-Evolving Agent Skills: Feedback Dynamics over Multiple Rounds [43.5891705224868]
  Self-evolving skill systems promise to improve agents by turning execution feedback into persistent skill updates without changing the underlying model.<n>We present a controlled evaluation framework across five benchmarks and three models.<n>Overall, persistent skill self-evolution is better understood as sparse, validation-filtered search with model- and benchmark-dependent returns.
  arXiv  Detail & Related papers  (2026-07-31T04:02:51Z)
• Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence [56.25095230687242]
  Coding agents often over-trust their own local validation routines and declare success on artifacts that satisfy surface checks.<n>This problem is especially acute in conversion, where prior evaluation is outcome-driven.<n>Across blind conversion attempts, the best system reaches 26.7--28.9% overall pass rate despite Spec pass rates up to 91.1%.<n>This suggests that failures stem more from contract-misaligned self-validation than from limited budget or backbone strength.
  arXiv  Detail & Related papers  (2026-05-27T19:57:15Z)
• Confidence and Calibration of Activation Oracles for Reliable Interpretation of Language Model Internals [2.4305775926851334]
  We investigate 6 different methods for estimating the confidence of activation oracles.<n> bootstrap mode frequency is the best-calibrated method among those tested.<n>The log-prob baseline can serve as a fast triage signal at a fraction of the cost.
  arXiv  Detail & Related papers  (2026-05-25T17:08:47Z)
• From Correlation to Cause: A Five-Stage Methodology for Feature Analysis in Transformer Language Models [0.0]
  We propose a five-stage methodology for causal feature analysis in transformer language models.<n>We demonstrate it end-to-end on GPT-2 small performing the Indirect Object Identification (IOI) task.
  arXiv  Detail & Related papers  (2026-05-21T13:25:16Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• Automated Self-Testing as a Quality Gate: Evidence-Driven Release Management for LLM Applications [51.56484100374058]
  We present an automated self-testing framework that introduces quality gates with evidence-based release decisions.<n>We evaluate the framework through a longitudinal case study of an internally deployed multi-agent conversational AI system.
  arXiv  Detail & Related papers  (2026-03-13T20:44:15Z)
• Aligning to Illusions: Choice Blindness in Human and AI Feedback [2.912535004465788]
  In a human choice blindness study, 91% of surreptitiously swapped preferences go undetected, extending choice blindness to third-person evaluative comparison of unfamiliar text.<n>We find detection relies on shallow text matching rather than genuine self-monitoring.<n>In a dose-response experiment across two architectures, one-sixth to one-third of labels must be corrupted before the reward signal halves.<n>A Best-of-N evaluation confirms this translates to downstream policy degradation.
  arXiv  Detail & Related papers  (2026-03-09T14:10:36Z)
• Shrinking the Generation-Verification Gap with Weak Verifiers [42.538675831498715]
  Verifiers can improve language model capabilities by scoring and ranking responses from generated candidates.<n>Weaver is a framework for designing a strong verifier by combining multiple weak, imperfect verifiers.
  arXiv  Detail & Related papers  (2025-06-22T23:38:15Z)
• A Framework for Creating Non-Regressive Test Cases via Branch Consistency Analysis Driven by Descriptions [9.141981611891715]
  DISTINCT is a Description-guided, branch-consistency analysis framework.<n>It transforms Large Language Model (LLM)-based generators into fault-aware test generators.<n>It achieves an average improvement of 14.64% in Compilation Success Rate (CSR) and 6.66% in Passing Rate (PR)
  arXiv  Detail & Related papers  (2025-06-09T07:05:48Z)
• Learning to Generate Unit Tests for Automated Debugging [52.63217175637201]
  Unit tests (UTs) play an instrumental role in assessing code correctness as well as providing feedback to large language models (LLMs)<n>We propose UTGen, which teaches LLMs to generate unit test inputs that reveal errors along with their correct expected outputs.<n>We show that UTGen outperforms other LLM-based baselines by 7.59% based on a metric measuring the presence of both error-revealing UT inputs and correct UT outputs.
  arXiv  Detail & Related papers  (2025-02-03T18:51:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.