Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs

• URL: http://arxiv.org/abs/2510.11059v1
• Date: Mon, 13 Oct 2025 06:49:28 GMT
• Title: Defects4C: Benchmarking Large Language Model Repair Capability with C/C++ Bugs
• Abstract summary: Automated Program Repair (APR) plays a critical role in enhancing the quality and reliability of software systems.<n>There remains a significant gap in research on C/C++ program repair, despite the widespread use of C/C++ and the prevalence of associated vulnerabilities.<n>We introduce Defects4C, a benchmark specifically designed for C/C++ program repair.
• Score: 22.51003705398642
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automated Program Repair (APR) plays a critical role in enhancing the quality and reliability of software systems. While substantial progress has been made in Java-based APR, largely facilitated by benchmarks like Defects4J, there remains a significant gap in research on C/C++ program repair, despite the widespread use of C/C++ and the prevalence of associated vulnerabilities. This gap is primarily due to the lack of high-quality, open-source benchmarks tailored for C/C++. To address this issue, we introduce Defects4C, a comprehensive and executable benchmark specifically designed for C/C++ program repair. Our dataset is constructed from real-world C/C++ repositories and includes a large collection of bug-relevant commits (9M in total), 248 high-quality buggy functions, and 102 vulnerable functions, all paired with test cases for reproduction. These resources enable rigorous evaluation of repair techniques and support the retraining of learning-based approaches for enhanced performance. Using Defects4C, we conduct a comprehensive empirical study evaluating the effectiveness of 24 state-of-the-art large language models (LLMs) in repairing C/C++ faults. Our findings offer valuable insights into the strengths and limitations of current LLM-based APR techniques in this domain, highlighting both the need for more robust methods and the critical role of Defects4C in advancing future research

Related papers

• BASICS: Binary Analysis and Stack Integrity Checker System for Buffer Overflow Mitigation [0.0]
  Cyber-Physical Systems have played an essential role in our daily lives, providing critical services such as power and water.<n>Traditional vulnerability discovery techniques struggle with scalability and precision when applied directly to the binary code of C programs.<n>This work introduces a novel approach designed to overcome these limitations by leveraging model checking and concolic execution techniques.
  arXiv  Detail & Related papers  (2025-11-24T20:11:41Z)
• QiMeng-NeuComBack: Self-Evolving Translation from IR to Assembly Code [52.66657751895655]
  Large Language Models (LLMs) offer a compelling new paradigm: Neural Compilation.<n>This paper introduces NeuComBack, a novel benchmark dataset specifically designed for IR-to-assembly compilation.<n>We propose a self-evolving prompt optimization method that enables LLMs to evolve their internal prompt strategies.
  arXiv  Detail & Related papers  (2025-11-03T03:20:26Z)
• CCrepairBench: A High-Fidelity Benchmark and Reinforcement Learning Framework for C++ Compilation Repair [18.624106902572155]
  We present CCrepair, a novel, large-scale C++ compilation error dataset constructed through a sophisticated generate-and-verify pipeline.<n>Second, we propose a Reinforcement Learning paradigm guided by a hybrid reward signal, shifting the focus from mere compilability to the semantic quality of the fix.
  arXiv  Detail & Related papers  (2025-09-19T07:06:27Z)
• Automated Repair of C Programs Using Large Language Models [0.0]
  This study explores the potential of Large Language Models (LLMs) in automating the repair of C programs.<n>We present a framework that integrates spectrum-based fault localization (SBFL), runtime feedback, and Chain-of-Thought-structured prompting into an autonomous repair loop.<n>Our approach achieves 44.93% repair accuracy, representing a 3.61% absolute improvement over strong state-of-the-art APR baselines.
  arXiv  Detail & Related papers  (2025-09-02T04:34:11Z)
• Refactoring $\neq$ Bug-Inducing: Improving Defect Prediction with Code Change Tactics Analysis [54.361900378970134]
  Just-in-time defect prediction (JIT-DP) aims to predict the likelihood of code changes resulting in software defects at an early stage.<n>Prior research has largely ignored code during both the evaluation and methodology phases, despite its prevalence.<n>We propose Code chAnge Tactics (CAT) analysis to categorize code and its propagation, which improves labeling accuracy in the JIT-Defects4J dataset by 13.7%.
  arXiv  Detail & Related papers  (2025-07-25T23:29:25Z)
• Specification-Guided Repair of Arithmetic Errors in Dafny Programs using LLMs [79.74676890436174]
  We present an APR tool for Dafny that uses formal specifications as oracles for fault localization and repair.<n>We localize faults through a series of steps, which include using Hoare logic to determine the state of each statement within the program.<n>Our tool achieves 89.6% fault localization coverage and GPT-4o mini yields the highest repair success rate of 74.18%.
  arXiv  Detail & Related papers  (2025-07-04T15:36:12Z)
• CCISolver: End-to-End Detection and Repair of Method-Level Code-Comment Inconsistency [33.30328162446649]
  Code-comment inconsistency (CCI) can negatively affect software development, testing, and maintenance.<n>We present an innovative end-to-end framework, CCIBench, designed to improve code quality by identifying and rectifying CCIs.
  arXiv  Detail & Related papers  (2025-06-25T15:56:07Z)
• SoK: Automated Vulnerability Repair: Methods, Tools, and Assessments [21.762329874797086]
  Manual vulnerability repair is labor-intensive and time-consuming because it relies on human experts.<n>We present the systematization of methods through the three steps of workflow: vulnerability analysis, patch generation, and patch validation.<n>We use Vul4C to evaluate seven tools for C/C++ programs and use the third-party Vul4J dataset to evaluate two tools for Java programs.
  arXiv  Detail & Related papers  (2025-06-13T11:59:04Z)
• ReLearn: Unlearning via Learning for Large Language Models [64.2802606302194]
  We propose ReLearn, a data augmentation and fine-tuning pipeline for effective unlearning.<n>This framework introduces Knowledge Forgetting Rate (KFR) and Knowledge Retention Rate (KRR) to measure knowledge-level preservation.<n>Our experiments show that ReLearn successfully achieves targeted forgetting while preserving high-quality output.
  arXiv  Detail & Related papers  (2025-02-16T16:31:00Z)
• How Far Can We Go with Practical Function-Level Program Repair? [11.71750828464698]
  This paper investigates the effect of few-shot learning mechanism and the auxiliary repair-relevant information on function-level APR. We propose an LLM-based function-level APR technique, namely SRepair, which adopts a dual-LLM framework to leverage the power of the auxiliary repair-relevant information.
  arXiv  Detail & Related papers  (2024-04-19T12:14:09Z)
• DebugBench: Evaluating Debugging Capability of Large Language Models [80.73121177868357]
  DebugBench is a benchmark for Large Language Models (LLMs) It covers four major bug categories and 18 minor types in C++, Java, and Python. We evaluate two commercial and four open-source models in a zero-shot scenario.
  arXiv  Detail & Related papers  (2024-01-09T15:46:38Z)
• Is Self-Repair a Silver Bullet for Code Generation? [68.02601393906083]
  Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair -- in which the model debugs and repairs its own code -- has recently become a popular way to boost performance. We analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS.
  arXiv  Detail & Related papers  (2023-06-16T15:13:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.