Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking

• URL: http://arxiv.org/abs/2606.29088v3
• Date: Wed, 01 Jul 2026 09:58:04 GMT
• Title: Diff-Based Code Corruption using LLMs for Large-Scale Bugfix Benchmarking
• Abstract summary: We introduce MegaBugFix, a large-scale bugfixing benchmark containing 12,629 buggy Python programs synthesized from correct ones by a Large Language Model.<n>We evaluate 13 open-weight models on MegaBugFix and baseline benchmarks, finding consistently lower performance on MegaBugFix.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: There are various benchmarks to evaluate bugfixing capabilities of Large Language Models. However, most widespread benchmarks do not fully reflect real-world bugfixing practices. They are small, weakening statistical reliability, and the buggy programs are often similar to one another, potentially distorting evaluation results. The range of bug types can also be narrow, failing to capture a representative range of bugs. To address these issues, we introduce MegaBugFix, a large-scale bugfixing benchmark containing 12,629 buggy Python programs synthesized from correct ones by a Large Language Model. Bug injections were generated as diffs representing code changes. Through this approach, we were able to avoid common pitfalls of LLM-based mutation techniques like injecting overly simplistic bugs or failing to modify the input program. We evaluated 13 open-weight models on MegaBugFix and baseline benchmarks, finding consistently lower performance on MegaBugFix. This reveals that our benchmark presents more challenging bugs and exposes model failures that may remain hidden when evaluating on existing benchmarks. The benchmark and fine-tuned model used for bug injection are available at hf.co/collections/szalontaib/megabugfix.

Related papers

• Model See, Model Do? Exposure-Aware Evaluation of Bug-vs-Fix Preference in Code LLMs [14.187824830583784]
  Distinguishing whether an LLM prefers correct code might be influenced by what it's been exposed to during training.<n>We introduce an exposure-aware evaluation framework that quantifies how prior exposure to buggy versus fixed code influences a model's preference.
  arXiv  Detail & Related papers  (2026-01-15T15:14:29Z)
• BugPilot: Complex Bug Generation for Efficient Learning of SWE Skills [59.003563837981886]
  High quality bugs are key to training the next generation of language model based software engineering (SWE) agents.<n>We introduce a novel method for synthetic generation of difficult and diverse bugs.
  arXiv  Detail & Related papers  (2025-10-22T17:58:56Z)
• How Many Code and Test Cases Are Enough? Evaluating Test Cases Generation from a Binary-Matrix Perspective [51.30005925128432]
  evaluating test cases automatically generated by Large Language Models (LLMs) is a critical yet challenging task.<n>Existing benchmarks suffer from high computational costs, score inflation, and a bias towards trivial bugs over rare, critical faults.<n>We introduce a framework that formalizes benchmark construction as finding an optimal diagnostic basis in a binary code-test matrix.
  arXiv  Detail & Related papers  (2025-10-09T18:29:24Z)
• Do Large Language Model Benchmarks Test Reliability? [66.1783478365998]
  We investigate how well current benchmarks quantify model reliability.<n>Motivated by this gap in the evaluation of reliability, we propose the concept of so-called platinum benchmarks.<n>We evaluate a wide range of models on these platinum benchmarks and find that, indeed, frontier LLMs still exhibit failures on simple tasks.
  arXiv  Detail & Related papers  (2025-02-05T18:58:19Z)
• Are Large Language Models Memorizing Bug Benchmarks? [6.640077652362016]
  Large Language Models (LLMs) have become integral to various software engineering tasks, including code generation, bug detection, and repair.<n>A growing concern within the software engineering community is that benchmarks may not reliably reflect true LLM performance due to the risk of data leakage.<n>We systematically evaluate popular LLMs to assess their susceptibility to data leakage from widely used bug benchmarks.
  arXiv  Detail & Related papers  (2024-11-20T13:46:04Z)
• From Code to Correctness: Closing the Last Mile of Code Generation with Hierarchical Debugging [7.503636246307676]
  We introduce Multi-Granularity Debugger (MG Debugger), a hierarchical code debugger by isolating, identifying, and resolving bugs at various levels of granularity.<n>MG Debugger decomposes problematic code into a hierarchical tree structure of subfunctions, with each level representing a particular granularity of error.<n>It achieves an 18.9% improvement in accuracy over seed generations in HumanEval and a 97.6% repair success rate in HumanEvalFix.
  arXiv  Detail & Related papers  (2024-10-02T03:57:21Z)
• The Fact Selection Problem in LLM-Based Program Repair [3.7005619077967133]
  We show that each fact, ranging from simple syntactic details like code context to semantic information previously unexplored in the context of Python projects, is beneficial. Importantly, we discovered that the effectiveness of program repair prompts is non-monotonic over the number of used facts. We develop a basic statistical model, named Maniple, which selects facts specific to a given bug to include in the prompt.
  arXiv  Detail & Related papers  (2024-04-08T13:41:32Z)
• DebugBench: Evaluating Debugging Capability of Large Language Models [80.73121177868357]
  DebugBench is a benchmark for Large Language Models (LLMs) It covers four major bug categories and 18 minor types in C++, Java, and Python. We evaluate two commercial and four open-source models in a zero-shot scenario.
  arXiv  Detail & Related papers  (2024-01-09T15:46:38Z)
• RAP-Gen: Retrieval-Augmented Patch Generation with CodeT5 for Automatic Program Repair [75.40584530380589]
  We propose a novel Retrieval-Augmented Patch Generation framework (RAP-Gen) RAP-Gen explicitly leveraging relevant fix patterns retrieved from a list of previous bug-fix pairs. We evaluate RAP-Gen on three benchmarks in two programming languages, including the TFix benchmark in JavaScript, and Code Refinement and Defects4J benchmarks in Java.
  arXiv  Detail & Related papers  (2023-09-12T08:52:56Z)
• Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectors [105.12462629663757]
  In this work, we aggregate factuality error annotations from nine existing datasets and stratify them according to the underlying summarization model. We compare performance of state-of-the-art factuality metrics, including recent ChatGPT-based metrics, on this stratified benchmark and show that their performance varies significantly across different types of summarization models.
  arXiv  Detail & Related papers  (2022-05-25T15:26:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.