ReFuzzer: Feedback-Driven Approach to Enhance Validity of LLM-Generated Test Programs

• URL: http://arxiv.org/abs/2508.03603v1
• Date: Tue, 05 Aug 2025 16:17:02 GMT
• Title: ReFuzzer: Feedback-Driven Approach to Enhance Validity of LLM-Generated Test Programs
• Abstract summary: Existing compiler fuzzers often produce syntactically or semantically invalid test programs.<n>We introduce ReFuzzer, a framework for refining LLM-generated test programs.<n>We evaluate ReFuzzer's effectiveness across black-, grey- and white-box fuzzing approaches targeting LLVM/Clang.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Existing LLM-based compiler fuzzers often produce syntactically or semantically invalid test programs, limiting their effectiveness in exercising compiler optimizations and backend components. We introduce ReFuzzer, a framework for refining LLM-generated test programs by systematically detecting and correcting compilation and runtime violations (e.g. division by zero or array out-of-bounds accesses). ReFuzzer employs a feedback loop with a local LLM to validate and filter erroneous programs before execution, improving fuzzing effectiveness beyond crash detection and enabling the generation of diverse yet valid test programs. We evaluated ReFuzzer's effectiveness across black-, grey- and white-box fuzzing approaches targeting LLVM/Clang. ReFuzzer improved test programs' validity from 47.0-49.4% to 96.6-97.3%, with an average processing time of 2.9-3.5 s per test program on a dual-GPU machine. Further, refuzzing significantly increased code coverage in critical optimization and IR generation components. For example, vectorization coverage had an absolute improvement of 9.2%, 2.3%, and 7.1% in black-, grey-, and white-box fuzzing, enhancing testing effectiveness.

Related papers

• Overcoming the Weakest-Link Effect in LLM-Driven Program Optimization via Heterogeneous Edit Recombination [59.679182823718556]
  HERO is a program that generates diverse, non-overlapping atomic edits and then systematically selects and composes them into coherent program improvements using evaluator scores.<n>We evaluate HERO across algorithmic problems, strategy games, the design of LLM-based agentic systems, and robotic path planning.
  arXiv  Detail & Related papers  (2026-07-31T02:05:45Z)
• LLM-as-a-Verifier: A General-Purpose Verification Framework [74.40111651545979]
  We introduce LLM-as-a-Verifier, a general-purpose verification framework.<n>It provides fine-grained feedback for agentic tasks without requiring additional training.<n>It achieves state-of-the-art performance on several benchmarks.
  arXiv  Detail & Related papers  (2026-07-06T17:59:35Z)
• SWE-Mutation: Can LLMs Generate Reliable Test Suites in Software Engineering? [32.40265738661726]
  Test suites are indispensable for synthesizing program repair trajectories and for providing precise feedback signals in reinforcement learning.<n>We introduce SWE-Mutation, a benchmark for evaluating LLM-generated test suites.<n>We propose an agentic, language-agnostic framework for automatically generating complex mutants.
  arXiv  Detail & Related papers  (2026-05-21T08:45:50Z)
• Evaluating LLM-Based Test Generation Under Software Evolution [7.140756378584939]
  Large Language Models (LLMs) are increasingly used for automated unit test generation.<n>We present a large-scale empirical study of LLM-based test generation under program changes.
  arXiv  Detail & Related papers  (2026-03-24T17:14:18Z)
• $\ abla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• CodeContests-O: Powering LLMs via Feedback-Driven Iterative Test Case Generation [71.42965967582147]
  Existing approaches attempt to synthesize test cases using Large Language Models (LLMs)<n>We propose a $textbfFeedback-Bench Iterative Framework$ for comprehensive test case construction.<n>Our dataset achieves an average True Positive Rate (TPR) of $89.37%$ and True Negative Rate (TNR) of $90.89%$, significantly outperforming the CodeContests and CodeContests+ by margins of $4.32%$ and $9.37%$, respectively.
  arXiv  Detail & Related papers  (2026-01-20T07:32:44Z)
• InspectCoder: Dynamic Analysis-Enabled Self Repair through interactive LLM-Debugger Collaboration [71.18377595277018]
  Large Language Models (LLMs) frequently generate buggy code with complex logic errors that are challenging to diagnose.<n>We present InspectCoder, the first agentic program repair system that empowers LLMs to actively conduct dynamic analysis via interactive debugger control.
  arXiv  Detail & Related papers  (2025-10-21T06:26:29Z)
• LLMs are All You Need? Improving Fuzz Testing for MOJO with Large Language Models [7.171282546185869]
  Large language models (LLMs) have revolutionized software testing, particularly fuzz testing, by automating the generation of diverse and effective test inputs.<n>MoJO is a high-performance AI programming language blending Python's usability with the efficiency of C and C++.<n>MoJOFuzzer is the first adaptive LLM-based fuzzing framework designed for zero-shot learning environments of emerging programming languages.
  arXiv  Detail & Related papers  (2025-10-11T11:37:18Z)
• BASFuzz: Towards Robustness Evaluation of LLM-based NLP Software via Automated Fuzz Testing [8.893978269498524]
  BASFuzz is an efficient fuzz testing method tailored for large language model (LLM)-based NLP software.<n>A Beam-Annealing Search algorithm, which integrates beam search and simulated annealing, is employed to design an efficient fuzzing loop.<n>Experiments demonstrate that BASFuzz achieves a testing effectiveness of 90.335% while reducing the average time overhead by 2,163.852 seconds.
  arXiv  Detail & Related papers  (2025-09-22T03:13:57Z)
• TestWeaver: Execution-aware, Feedback-driven Regression Testing Generation with Large Language Models [5.871736617580623]
  Regression testing ensures that code changes do not unintentionally break existing functionality.<n>Recent advances in large language models (LLMs) have shown promise in automating test generation for regression testing.<n>We present TestWeaver, a novel approach that integrates lightweight program analysis to guide test generation more effectively.
  arXiv  Detail & Related papers  (2025-08-02T08:13:02Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• Input Reduction Enhanced LLM-based Program Repair [2.098274800451098]
  Test inputs are crucial for reasoning the root cause of failures.<n>When the test inputs are extensive in the prompt, this may trigger the "lost-in-the-middle" issue, compromising repair performance.<n>We propose ReduceFix, an APR approach with a built-in component that automatically reduces test inputs while retaining their failure-inducing behavior.
  arXiv  Detail & Related papers  (2025-07-21T05:26:32Z)
• Fuzzing-based Mutation Testing of C/C++ Software in Cyber-Physical Systems [2.362412515574206]
  State-of-the-art mutation testing techniques for C and C++ software depend on symbolic execution.<n>We propose relying on fuzz testing, which has demonstrated its effectiveness for C and C++ software.
  arXiv  Detail & Related papers  (2025-03-31T13:55:27Z)
• PerfCodeGen: Improving Performance of LLM Generated Code with Execution Feedback [78.89596149768458]
  Large Language Models (LLMs) are widely adopted for assisting in software development tasks.<n>We propose PerfCodeGen, a training-free framework that enhances the performance of LLM-generated code.
  arXiv  Detail & Related papers  (2024-11-18T06:22:38Z)
• AIME: AI System Optimization via Multiple LLM Evaluators [79.03422337674664]
  AIME is an evaluation protocol that utilizes multiple LLMs that each independently generate an evaluation on separate criteria and then combine them via concatenation. We show AIME outperforming baseline methods in code generation tasks, with up to $62%$ higher error detection rate and up to $16%$ higher success rate than a single LLM evaluation protocol on LeetCodeHard and HumanEval datasets.
  arXiv  Detail & Related papers  (2024-10-04T04:03:24Z)
• WhiteFox: White-Box Compiler Fuzzing Empowered by Large Language Models [11.33856613057612]
  We propose WhiteFox, the first white-box compiler fuzzer using Large Language Models with source-code information. WhiteFox can generate high-quality test programs to exercise deep optimizations, practicing up to 8X more than state-of-the-art fuzzers. WhiteFox has found 101 bugs for the DL compilers, with 92 confirmed as previously unknown and 70 fixed.
  arXiv  Detail & Related papers  (2023-10-24T16:39:06Z)
• Effective Test Generation Using Pre-trained Large Language Models and Mutation Testing [13.743062498008555]
  We introduce MuTAP for improving the effectiveness of test cases generated by Large Language Models (LLMs) in terms of revealing bugs. MuTAP is capable of generating effective test cases in the absence of natural language descriptions of the Program Under Test (PUTs) Our results show that our proposed method is able to detect up to 28% more faulty human-written code snippets.
  arXiv  Detail & Related papers  (2023-08-31T08:48:31Z)
• LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond [135.8013388183257]
  We propose a new protocol for inconsistency detection benchmark creation and implement it in a 10-domain benchmark called SummEdits. Most LLMs struggle on SummEdits, with performance close to random chance. The best-performing model, GPT-4, is still 8% below estimated human performance.
  arXiv  Detail & Related papers  (2023-05-23T21:50:06Z)
• LEVER: Learning to Verify Language-to-Code Generation with Execution [64.36459105535]
  We propose LEVER, a simple approach to improve language-to-code generation by learning to verify the generated programs with their execution results. Specifically, we train verifiers to determine whether a program sampled from the LLMs is correct or not based on the natural language input, the program itself and its execution results. LEVER consistently improves over the base code LLMs(4.6% to 10.9% with code-davinci) and achieves new state-of-the-art results on all of them.
  arXiv  Detail & Related papers  (2023-02-16T18:23:22Z)
• Learning Performance-Improving Code Edits [107.21538852090208]
  We introduce a framework for adapting large language models (LLMs) to high-level program optimization. First, we curate a dataset of performance-improving edits made by human programmers of over 77,000 competitive C++ programming submission pairs. For prompting, we propose retrieval-based few-shot prompting and chain-of-thought, and for finetuning, these include performance-conditioned generation and synthetic data augmentation based on self-play.
  arXiv  Detail & Related papers  (2023-02-15T18:59:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.