PatchGuru: Patch Oracle Inference from Natural Language Artifacts with Large Language Models

• URL: http://arxiv.org/abs/2602.05270v1
• Date: Thu, 05 Feb 2026 03:48:17 GMT
• Title: PatchGuru: Patch Oracle Inference from Natural Language Artifacts with Large Language Models
• Authors: Thanh Le-Cong, Bach Le, Toby Murray, Michael Pradel, Cristian Cadar,
• Abstract summary: We present PatchGuru, the first automated technique that infers executable patch specifications from real-world pull requests (PRs)<n>Given a PR, PatchGuru uses large language models (LLMs) to extract developer intent from NL artifacts and synthesizes patch oracles.<n>PatchGuru iteratively refines inferred oracles by comparing pre- and post-patch behaviors, identifies violations, filters inconsistencies via self-review, and generates bug reports.
• Score: 16.633386478990037
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As software systems evolve, patches may unintentionally alter program behavior. Validating patches against their intended semantics is difficult due to incomplete regression tests and informal, non-executable natural language (NL) descriptions of patch intent. We present PatchGuru, the first automated technique that infers executable patch specifications from real-world pull requests (PRs). Given a PR, PatchGuru uses large language models (LLMs) to extract developer intent from NL artifacts and synthesizes patch oracles: under-approximate yet practical specifications expressed as runtime assertions in comparison programs that integrate pre- and post-patch versions. Patch oracles focus on patch-relevant behaviors, enable automated validation, and support cross-version properties. PatchGuru iteratively refines inferred oracles by comparing pre- and post-patch behaviors, identifies violations, filters inconsistencies via self-review, and generates bug reports. We evaluate PatchGuru on 400 recent PRs from four widely used open-source Python projects. PatchGuru reports 39 warnings with a precision of 0.62, yielding 24 confirmed true positives, including 12 previously unknown bugs, 11 of which were subsequently fixed by developers. Compared to the state-of-the-art technique Testora, PatchGuru detects 17 more bugs (24 vs. 7) while improving precision from 0.32 to 0.62. PatchGuru incurs an average cost of 8.9 minutes and USD 0.07 per PR. These results suggest that PatchGuru complements code review and regression testing by providing executable documentation and automated validation of patch intent.

Related papers

• Historian: Reducing Manual Validation in APR Benchmarking via Evidence-Based Assessment [0.19853810231896352]
  We present Historian, a framework that leverages Large Language Models to perform multi-reference comparisons against a knowledge base of historically validated patches.<n>In leave-one-tool-out evaluation, Historian achieves 95.0% coverage with 88.4% accuracy, reducing manual validation to 5% of patches.
  arXiv  Detail & Related papers  (2026-02-28T13:41:29Z)
• BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills [59.003563837981886]
  High quality bugs are key to training the next generation of language model based software engineering (SWE) agents.<n>We introduce a novel method for synthetic generation of difficult and diverse bugs.
  arXiv  Detail & Related papers  (2025-10-22T17:58:56Z)
• What Do They Fix? LLM-Aided Categorization of Security Patches for Critical Memory Bugs [46.325755802511026]
  We developLM, a dual-method pipeline that integrates two approaches based on a Large Language Model (LLM) and a fine-tuned small language model.<n>LM successfully identified 111 of 5,140 recent Linux kernel patches addressing OOB or UAF vulnerabilities, with 90 true positives confirmed by manual verification.
  arXiv  Detail & Related papers  (2025-09-26T18:06:36Z)
• Are "Solved Issues" in SWE-bench Really Solved Correctly? An Empirical Study [18.117047833029073]
  Most popular benchmarks for automated issue solving are SWE-bench and its human-filtered subset SWE-bench Verified.<n>This paper presents an in-depth empirical study of the correctness of plausible patches generated by three state-of-the-art issue-solving tools evaluated on SWE-bench Verified.
  arXiv  Detail & Related papers  (2025-03-19T14:02:21Z)
• SOPBench: Evaluating Language Agents at Following Standard Operating Procedures and Constraints [59.645885492637845]
  SOPBench is an evaluation pipeline that transforms each service-specific SOP code program into a directed graph of executable functions.<n>Our approach transforms each service-specific SOP code program into a directed graph of executable functions and requires agents to call these functions based on natural language SOP descriptions.<n>We evaluate 18 leading models, and results show the task is challenging even for top-tier models.
  arXiv  Detail & Related papers  (2025-03-11T17:53:02Z)
• Assessing the Latent Automated Program Repair Capabilities of Large Language Models using Round-Trip Translation [44.3761164214368]
  We investigate Round-Trip Translation (RTT): translating code from one programming language into another programming or natural language and back.<n>We perform a detailed quantitative and qualitative analysis of RTT-generated patches in Java.<n>We find that RTT through English generates plausible patches for 100 of 164 bugs with GPT-4 on the HumanEval-Java benchmark, and 97 are found to be correct in our manual assessment.
  arXiv  Detail & Related papers  (2024-01-15T22:36:31Z)
• RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair [75.40584530380589]
  We propose a novel Retrieval-Augmented Patch Generation framework (RAP-Gen) RAP-Gen explicitly leveraging relevant fix patterns retrieved from a list of previous bug-fix pairs. We evaluate RAP-Gen on three benchmarks in two programming languages, including the TFix benchmark in JavaScript, and Code Refinement and Defects4J benchmarks in Java.
  arXiv  Detail & Related papers  (2023-09-12T08:52:56Z)
• Fixing Model Bugs with Natural Language Patches [38.67529353406759]
  We explore natural language patches that allow developers to provide corrective feedback at the right level of abstraction. We show that with a small amount of synthetic data, we can teach models to effectively use real patches on real data. We also show that finetuning on as many as 100 labeled examples may be needed to match the performance of a small set of language patches.
  arXiv  Detail & Related papers  (2022-11-07T05:49:19Z)
• Is this Change the Answer to that Problem? Correlating Descriptions of Bug and Code Changes for Evaluating Patch Correctness [8.606215760860362]
  We turn the patch correctness assessment into a Question Answering problem. We consider as inputs the bug reports as well as the natural language description of the generated patches. Experiments show that Quatrain can achieve an AUC of 0.886 on predicting patch correctness.
  arXiv  Detail & Related papers  (2022-08-08T13:32:58Z)
• Checking Patch Behaviour against Test Specification [4.723400023753107]
  We propose a hypothesis on how the link between the patch behaviour and failing test specifications can be drawn. We then propose BATS, an unsupervised learning-based system to predict patch correctness.
  arXiv  Detail & Related papers  (2021-07-28T11:39:06Z)
• (De)Randomized Smoothing for Certifiable Defense against Patch Attacks [136.79415677706612]
  We introduce a certifiable defense against patch attacks that guarantees for a given image and patch attack size. Our method is related to the broad class of randomized smoothing robustness schemes. Our results effectively establish a new state-of-the-art of certifiable defense against patch attacks on CIFAR-10 and ImageNet.
  arXiv  Detail & Related papers  (2020-02-25T08:39:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.