Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation

• URL: http://arxiv.org/abs/2310.02368v2
• Date: Mon, 06 Jan 2025 16:41:09 GMT
• Title: Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation
• Authors: Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, Alexey Svyatkovskiy,
• Abstract summary: Large Language Models (LLMs) have gained popularity for code generation, including the automated creation of test cases. LLMs are often trained on vast amounts of publicly available code, which may include test cases that do not adhere to best practices. We propose a novel technique called Reinforcement Learning from Static Quality Metrics (RLSQM)
• Score: 12.503002900186997
• License:
• Abstract: Software testing is a crucial aspect of software development, and the creation of high-quality tests that adhere to best practices is essential for effective maintenance. Recently, Large Language Models (LLMs) have gained popularity for code generation, including the automated creation of test cases. However, these LLMs are often trained on vast amounts of publicly available code, which may include test cases that do not adhere to best practices and may even contain test smells (anti-patterns). To address this issue, we propose a novel technique called Reinforcement Learning from Static Quality Metrics (RLSQM). To begin, we analyze the anti-patterns generated by the LLM and show that LLMs can generate undesirable test smells. Thus, we train specific reward models for each static quality metric, then utilize Proximal Policy Optimization (PPO) to train models for optimizing a single quality metric at a time. Furthermore, we amalgamate these rewards into a unified reward model aimed at capturing different best practices and quality aspects of tests. By comparing RL-trained models with those trained using supervised learning, we provide insights into how reliably utilize RL to improve test generation quality and into the effects of various training strategies. Our experimental results demonstrate that the RL-optimized model consistently generated high-quality test cases compared to the base LLM, improving the model by up to 21%, and successfully generates nearly 100% syntactically correct code. RLSQM also outperformed GPT-4 on four out of seven metrics. This represents a significant step towards enhancing the overall efficiency and reliability of software testing through Reinforcement Learning and static quality metrics. Our data are available at https://figshare.com/s/ded476c8d4c221222849.

Related papers

• UnitCoder: Scalable Iterative Code Synthesis with Unit Test Guidance [65.01483640267885]
  Large Language Models (LLMs) have demonstrated remarkable capabilities in various tasks, yet code generation remains a major challenge. We introduce UnitCoder, a systematic pipeline leveraging model-generated unit tests to guide and validate the code generation process. Our work presents a scalable approach that leverages model-generated unit tests to guide the synthesis of high-quality code data from pre-training corpora.
  arXiv  Detail & Related papers  (2025-02-17T05:37:02Z)
• Clear Preferences Leave Traces: Reference Model-Guided Sampling for Preference Learning [59.11519451499754]
  Direct Preference Optimization (DPO) has emerged as a de-facto approach for aligning language models with human preferences. Recent work has shown DPO's effectiveness relies on training data quality. We discover that reference model probability space naturally detects high-quality training samples.
  arXiv  Detail & Related papers  (2025-01-25T07:21:50Z)
• Dynamic Scaling of Unit Tests for Code Reward Modeling [27.349232888627558]
  Current large language models (LLMs) often struggle to produce accurate responses on the first attempt for complex reasoning tasks like code generation. We propose CodeRM-8B, a lightweight yet effective unit test generator that enables efficient and high-quality unit test scaling.
  arXiv  Detail & Related papers  (2025-01-02T04:33:31Z)
• Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation [12.503002900186997]
  Large Language Models (LLMs) have gained popularity for automated test case generation. Because LLMs are trained on vast amounts of open-source code, they often generate test cases that do not adhere to best practices. We propose Reinforcement Learning from Static Quality Metrics (RLSQM) to generate high-quality unit tests based on static analysis-based quality metrics.
  arXiv  Detail & Related papers  (2024-12-18T20:20:01Z)
• Star-Agents: Automatic Data Optimization with LLM Agents for Instruction Tuning [71.2981957820888]
  We propose a novel Star-Agents framework, which automates the enhancement of data quality across datasets. The framework initially generates diverse instruction data with multiple LLM agents through a bespoke sampling method. The generated data undergo a rigorous evaluation using a dual-model method that assesses both difficulty and quality.
  arXiv  Detail & Related papers  (2024-11-21T02:30:53Z)
• Large-scale, Independent and Comprehensive study of the power of LLMs for test case generation [11.056044348209483]
  Unit testing, crucial for identifying bugs in code modules like classes and methods, is often neglected by developers due to time constraints. Large Language Models (LLMs), like GPT and Mistral, show promise in software engineering, including in test generation.
  arXiv  Detail & Related papers  (2024-06-28T20:38:41Z)
• Uncertainty Aware Learning for Language Model Alignment [97.36361196793929]
  We propose uncertainty-aware learning (UAL) to improve the model alignment of different task scenarios. We implement UAL in a simple fashion -- adaptively setting the label smoothing value of training according to the uncertainty of individual samples. Experiments on widely used benchmarks demonstrate that our UAL significantly and consistently outperforms standard supervised fine-tuning.
  arXiv  Detail & Related papers  (2024-06-07T11:37:45Z)
• Code-Aware Prompting: A study of Coverage Guided Test Generation in Regression Setting using LLM [32.44432906540792]
  We present SymPrompt, a code-aware prompting strategy for large language models in test generation. SymPrompt enhances correct test generations by a factor of 5 and bolsters relative coverage by 26% for CodeGen2. Notably, when applied to GPT-4, SymPrompt improves coverage by over 2x compared to baseline prompting strategies.
  arXiv  Detail & Related papers  (2024-01-31T18:21:49Z)
• Automatic Unit Test Data Generation and Actor-Critic Reinforcement Learning for Code Synthesis [16.88062487980405]
  We present a novel approach to automatically obtain data consisting of function signatures and associated Unit Tests. We show that it, in conjunction with automatically generated training data, leads to improvement of a pre-trained code language model's performance.
  arXiv  Detail & Related papers  (2023-10-20T17:13:16Z)
• CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning [92.36705236706678]
  "CodeRL" is a new framework for program synthesis tasks through pretrained LMs and deep reinforcement learning. During inference, we introduce a new generation procedure with a critical sampling strategy. For the model backbones, we extended the encoder-decoder architecture of CodeT5 with enhanced learning objectives.
  arXiv  Detail & Related papers  (2022-07-05T02:42:15Z)
• Text Generation with Efficient (Soft) Q-Learning [91.47743595382758]
  Reinforcement learning (RL) offers a more flexible solution by allowing users to plug in arbitrary task metrics as reward. We introduce a new RL formulation for text generation from the soft Q-learning perspective. We apply the approach to a wide range of tasks, including learning from noisy/negative examples, adversarial attacks, and prompt generation.
  arXiv  Detail & Related papers  (2021-06-14T18:48:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.