Agentic Bug Reproduction for Effective Automated Program Repair at Google

• URL: http://arxiv.org/abs/2502.01821v2
• Date: Tue, 11 Mar 2025 02:30:46 GMT
• Title: Agentic Bug Reproduction for Effective Automated Program Repair at Google
• Abstract summary: This paper investigates automated BRT generation within an industry setting, specifically at Google.<n>We adapt and evaluate a state-of-the-art BRT generation technique, LIBRO, and present our agent-based approach, BRT Agent.<n>Our results show that providing BRTs to the APR system results in 30% more bugs with plausible fixes.
• Score: 9.64193881099048
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Bug reports often lack sufficient detail for developers to reproduce and fix the underlying defects. Bug Reproduction Tests (BRTs), tests that fail when the bug is present and pass when it has been resolved, are crucial for debugging, but they are rarely included in bug reports, both in open-source and in industrial settings. Thus, automatically generating BRTs from bug reports has the potential to accelerate the debugging process and lower time to repair. This paper investigates automated BRT generation within an industry setting, specifically at Google, focusing on the challenges of a large-scale, proprietary codebase and considering real-world industry bugs extracted from Google's internal issue tracker. We adapt and evaluate a state-of-the-art BRT generation technique, LIBRO, and present our agent-based approach, BRT Agent, which makes use of a fine-tuned Large Language Model (LLM) for code editing. Our BRT Agent significantly outperforms LIBRO, achieving a 28% plausible BRT generation rate, compared to 10% by LIBRO, on 80 human-reported bugs from Google's internal issue tracker. We further investigate the practical value of generated BRTs by integrating them with an Automated Program Repair (APR) system at Google. Our results show that providing BRTs to the APR system results in 30% more bugs with plausible fixes. Additionally, we introduce Ensemble Pass Rate (EPR), a metric which leverages the generated BRTs to select the most promising fixes from all fixes generated by APR system. Our evaluation on EPR for Top-K and threshold-based fix selections demonstrates promising results and trade-offs. For example, EPR correctly selects a plausible fix from a pool of 20 candidates in 70% of cases, based on its top-1 ranking.

Related papers

• Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes [55.114939648705395]
  Large language models (LLMs) have made automated program repair increasingly practical for real-world bugs.<n>Bug reproduction tests (BRTs) help close this gap by turning a bug report into an executable, bug-specific signal.<n>We propose CoHarden, a co-generation framework that uses the Lax signal as an in-loop convergence criterion.
  arXiv  Detail & Related papers  (2026-07-22T07:30:07Z)
• Anchored Self-Play for Code Repair [68.0728003844994]
  Code repair is an important capability for language models (LMs)<n>We propose __generator--fixer self-play__, in which a single model is trained with reinforcement learning to generate bugs and fix them.<n>On BugSourceBench, we find that self-play drifts toward difficult but unrealistic bugs.<n>We propose Anchored Self-Play (ASP), which anchors self-play with a small reference set by adding a code-embedding similarity reward for generation and mixing reference bugs into fixer training.
  arXiv  Detail & Related papers  (2026-07-03T17:51:52Z)
• SWE-Doctor: Guiding Software Engineering Agents with Runtime Diagnosis from Multi-Faceted Bug Reproduction Tests [18.192718828200782]
  We propose SWE-Doctor, a software issue resolution agent that guides patch generation with runtime diagnoses derived from multi-faceted BRT executions.<n>SWE-Doctor consistently outperforms existing agents across all 10 LLM-benchmark combinations.
  arXiv  Detail & Related papers  (2026-07-01T14:27:12Z)
• ReproRepo: Scaling Reproducibility Audits with GitHub Repository Issues [54.98461672730087]
  We introduce ReproRepo, a scalable framework for evaluation that leverages human-raised GitHub issues as naturally occurring supervision on realistic reproduction blockers.<n>We instantiate ReproRepo on 1,149 recent machine learning papers from major conferences and evaluate four frontier model-agent configurations.<n>Our results show that LLM agents, even without executing code, can identify many real-world problems from paper-repository pairs.
  arXiv  Detail & Related papers  (2026-06-16T17:58:05Z)
• Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
  Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
  arXiv  Detail & Related papers  (2026-02-02T19:06:15Z)
• Dynamic Cogeneration of Bug Reproduction Test in Agentic Program Repair [4.994693185232122]
  Bug Reproduction Tests (BRTs) have been used in many agentic Automated Program Repair (APR) systems.<n>In practice, when developers submit a patch, they often implement the BRT alongside the fix.<n>Cogeneration allows the APR agent to generate BRTs for at least as many bugs as a dedicated BRT agent.
  arXiv  Detail & Related papers  (2026-01-27T01:02:57Z)
• DoVer: Intervention-Driven Auto Debugging for LLM Multi-Agent Systems [48.971606069204825]
  DoVer is an intervention-driven debug framework for large language model (LLM)-based multi-agent systems.<n>It augments hypothesis generation with active verification through targeted interventions.<n>DoVer flips 18-28% of failed trials into successes, achieves up to 16% milestone progress, and validates or refutes 30-60% of failure hypotheses.
  arXiv  Detail & Related papers  (2025-12-07T09:23:48Z)
• BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills [59.003563837981886]
  High quality bugs are key to training the next generation of language model based software engineering (SWE) agents.<n>We introduce a novel method for synthetic generation of difficult and diverse bugs.
  arXiv  Detail & Related papers  (2025-10-22T17:58:56Z)
• REFINE: Enhancing Program Repair Agents through Context-Aware Patch Refinement [12.995571513415905]
  Large Language Models (LLMs) have recently shown strong potential in automatic program repair (APR)<n>LLMs often struggle to produce correct fixes due to limited understanding of code context and over-reliance on incomplete test suites.<n>We propose a novel patch refinement framework, Refine, that systematically transforms Draft Patches into correct ones.
  arXiv  Detail & Related papers  (2025-10-04T00:34:32Z)
• Abstain and Validate: A Dual-LLM Policy for Reducing Noise in Agentic Program Repair [7.118712516789191]
  Agentic Automated Program Repair (APR) is increasingly tackling complex, repository-level bugs in industry.<n>Showing unlikely patches to developers can lead to substantial noise.<n>We introduce two complementary policies to reduce such noise: bug abstention and patch validation policies.
  arXiv  Detail & Related papers  (2025-10-03T17:53:28Z)
• Where LLM Agents Fail and How They can Learn From Failures [62.196870049524364]
  Large Language Model (LLM) agents have shown promise in solving complex, multi-step tasks.<n>They amplify vulnerability to cascading failures, where a single root-cause error propagates through subsequent decisions.<n>Current systems lack a framework that can comprehensively understand agent error in a modular and systemic way.<n>We introduce the AgentErrorTaxonomy, a modular classification of failure modes spanning memory, reflection, planning, action, and system-level operations.
  arXiv  Detail & Related papers  (2025-09-29T18:20:27Z)
• Repair-R1: Better Test Before Repair [2.982543556561469]
  APR aims to automatically locate program defects, generate patches and validate the repairs.<n>Current APR methods typically utilize test cases only during the inference stage.<n>We propose Repair-R1, which introduces test cases into the model's training phase and shifts test generation to precede repair.
  arXiv  Detail & Related papers  (2025-07-30T17:24:05Z)
• Do AI models help produce verified bug fixes? [62.985237003585674]
  Large Language Models are used to produce corrections to software bugs.<n>This paper investigates how programmers use Large Language Models to complement their own skills.<n>The results are a first step towards a proper role for AI and LLMs in providing guaranteed-correct fixes to program bugs.
  arXiv  Detail & Related papers  (2025-07-21T17:30:16Z)
• Evaluating Agent-based Program Repair at Google [9.62742759337993]
  Agent-based program repair offers to automatically resolve complex bugs end-to-end.<n>Recent work has explored the use of agent-based repair approaches on the popular open-source SWE-Bench.<n>This paper explores the viability of using an agentic approach to address bugs in an enterprise context.
  arXiv  Detail & Related papers  (2025-01-13T18:09:25Z)
• A Unified Debugging Approach via LLM-Based Multi-Agent Synergy [39.11825182386288]
  FixAgent is an end-to-end framework for unified debug through multi-agent synergy. It significantly outperforms state-of-the-art repair methods, fixing 1.25$times$ to 2.56$times$ bugs on the repo-level benchmark, Defects4J.
  arXiv  Detail & Related papers  (2024-04-26T04:55:35Z)
• ContrastRepair: Enhancing Conversation-Based Automated Program Repair via Contrastive Test Case Pairs [23.419180504723546]
  ContrastRepair is a novel APR approach that augments conversation-driven APR by providing contrastive test pairs. We evaluate ContrastRepair on multiple benchmark datasets, including Defects4j, QuixBugs, and HumanEval-Java.
  arXiv  Detail & Related papers  (2024-03-04T12:15:28Z)
• A Novel Approach for Automatic Program Repair using Round-Trip Translation with Large Language Models [50.86686630756207]
  Research shows that grammatical mistakes in a sentence can be corrected by translating it to another language and back. Current generative models for Automatic Program Repair (APR) are pre-trained on source code and fine-tuned for repair. This paper proposes bypassing the fine-tuning step and using Round-Trip Translation (RTT): translation of code from one programming language to another programming or natural language, and back.
  arXiv  Detail & Related papers  (2024-01-15T22:36:31Z)
• RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair [75.40584530380589]
  We propose a novel Retrieval-Augmented Patch Generation framework (RAP-Gen) RAP-Gen explicitly leveraging relevant fix patterns retrieved from a list of previous bug-fix pairs. We evaluate RAP-Gen on three benchmarks in two programming languages, including the TFix benchmark in JavaScript, and Code Refinement and Defects4J benchmarks in Java.
  arXiv  Detail & Related papers  (2023-09-12T08:52:56Z)
• Prompting Is All You Need: Automated Android Bug Replay with Large Language Models [28.69675481931385]
  We propose AdbGPT, a new lightweight approach to automatically reproduce the bugs from bug reports through prompt engineering. AdbGPT leverages few-shot learning and chain-of-thought reasoning to elicit human knowledge and logical reasoning from LLMs. Our evaluations demonstrate the effectiveness and efficiency of our AdbGPT to reproduce 81.3% of bug reports in 253.6 seconds.
  arXiv  Detail & Related papers  (2023-06-03T03:03:52Z)
• Teaching Large Language Models to Self-Debug [62.424077000154945]
  Large language models (LLMs) have achieved impressive performance on code generation. We propose Self- Debugging, which teaches a large language model to debug its predicted program via few-shot demonstrations.
  arXiv  Detail & Related papers  (2023-04-11T10:43:43Z)
• Improving Automated Program Repair with Domain Adaptation [0.0]
  Automated Program Repair (APR) is defined as the process of fixing a bug/defect in the source code, by an automated tool. APR tools have recently experienced promising results by leveraging state-of-the-art Neural Language Processing (NLP) techniques.
  arXiv  Detail & Related papers  (2022-12-21T23:52:09Z)
• Large Language Models are Few-shot Testers: Exploring LLM-based General Bug Reproduction [14.444294152595429]
  The number of tests added in open source repositories due to issues was about 28% of the corresponding project test suite size. We propose LIBRO, a framework that uses Large Language Models (LLMs), which have been shown to be capable of performing code-related tasks. Our evaluation of LIBRO shows that, on the widely studied Defects4J benchmark, LIBRO can generate failure reproducing test cases for 33% of all studied cases.
  arXiv  Detail & Related papers  (2022-09-23T10:50:47Z)
• BigIssue: A Realistic Bug Localization Benchmark [89.8240118116093]
  BigIssue is a benchmark for realistic bug localization. We provide a general benchmark with a diversity of real and synthetic Java bugs. We hope to advance the state of the art in bug localization, in turn improving APR performance and increasing its applicability to the modern development cycle.
  arXiv  Detail & Related papers  (2022-07-21T20:17:53Z)
• SUPERNOVA: Automating Test Selection and Defect Prevention in AAA Video Games Using Risk Based Testing and Machine Learning [62.997667081978825]
  Testing video games is an increasingly difficult task as traditional methods fail to scale with growing software systems. We present SUPERNOVA, a system responsible for test selection and defect prevention while also functioning as an automation hub. The direct impact of this has been observed to be a reduction in 55% or more testing hours for an undisclosed sports game title.
  arXiv  Detail & Related papers  (2022-03-10T00:47:46Z)
• Break-It-Fix-It: Unsupervised Learning for Program Repair [90.55497679266442]
  We propose a new training approach, Break-It-Fix-It (BIFI), which has two key ideas. We use the critic to check a fixer's output on real bad inputs and add good (fixed) outputs to the training data. Based on these ideas, we iteratively update the breaker and the fixer while using them in conjunction to generate more paired data. BIFI outperforms existing methods, obtaining 90.5% repair accuracy on GitHub-Python and 71.7% on DeepFix.
  arXiv  Detail & Related papers  (2021-06-11T20:31:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.