Related papers: Benchmarking Large Language Models on Repairing Qiskit Programs using Bugs4Q

Benchmarking Large Language Models on Repairing Qiskit Programs using Bugs4Q

URL: http://arxiv.org/abs/2607.09007v1
Date: Fri, 10 Jul 2026 00:23:47 GMT
Title: Benchmarking Large Language Models on Repairing Qiskit Programs using Bugs4Q
Abstract summary: We evaluate two Bugs4Q versions containing 67 unique real Qiskit defects.<n>We find that quantum benchmarks can suffer from silent label errors when reference fixes stop executing or buggy programs no longer reproduce failures.
Score: 3.9289101676026146
License: http://creativecommons.org/licenses/by-nc-nd/4.0/
Abstract: In quantum programs, Bugs4Q is a widely used benchmark containing real quantum defects. However, its evaluation assumes that benchmark labels remain valid and that generated fixes execute in the target environment. We evaluate two Bugs4Q versions containing 67 unique real Qiskit defects, adding executable tests where missing, and re-run all entries across six pinned Qiskit releases (0.25.0, 0.45.0, 1.0.0, 1.1.1, 2.0.0, and 2.3.1). We find that quantum benchmarks can suffer from silent label inversion: entries become invalid without errors when reference fixes stop executing or buggy programs no longer reproduce failures. Thus, correctness depends on the (benchmark, version) pair rather than the benchmark alone. We evaluate four LLMs (GPT-4o-mini, GPT-5o-mini, GPT-5.4, and GPT-5.4-mini), generating up to 10 repair candidates per defect and testing them across all versions. GPT-5.4 achieves the highest pass@10 (48.8%), followed by GPT-5.4-mini (47.3%), GPT-5o-mini (30.3%), and GPT-4o-mini (22.6%). All models perform best on Qiskit 0.45.0 and decline after the Qiskit 1.0 transition. Many failures arise from deprecated or incompatible APIs rather than incorrect repairs, and 64\% of successful repairs occur on entries invalid under the target version. We release a re-validated, version-pinned Bugs4Q benchmark and show that benchmark validation must precede repair evaluation.

Related papers

MemToC: Benchmarking Memory-Tool Conflict Resolution in Large Language Models [49.47300047353726]
We introduce MemToC, a benchmark for post-tool-return arbitration with executable tools.<n>MemToC comprises 6,504 evaluation episodes constructed from 542 quality-controlled factual questions.<n>Across five open-weight 7-9B models, tool returns strongly dominate elicited closed-book answers.
arXiv Detail & Related papers (2026-08-26T18:22:03Z)
Beyond Fail-to-Pass: Iterative Hardening of Co-Generated Bug Reproduction Tests and Fixes [55.114939648705395]
Large language models (LLMs) have made automated program repair increasingly practical for real-world bugs.<n>Bug reproduction tests (BRTs) help close this gap by turning a bug report into an executable, bug-specific signal.<n>We propose CoHarden, a co-generation framework that uses the Lax signal as an in-loop convergence criterion.
arXiv Detail & Related papers (2026-07-22T07:30:07Z)
Benchmarking API Drift in LLM-Generated Quantum Code Across Successive SDK Versions [0.0]
Large language models can generate plausible quantum code, but it is unclear whether they can reliably target the specific software development kit (SDK) version requested by the user.<n>We introduce quantum-api-drift, a benchmark for measuring version fidelity.<n>We instantiate the benchmark with Qiskit, a representative quantum SDK that underwent substantial interface changes across v0.43, v1.3, and v2.0.
arXiv Detail & Related papers (2026-07-05T01:16:12Z)
On the Reproducibility of Quantum Software Defect Datasets: A Case Study of Bugs4Q [0.3383208046995158]
We conduct a replication study of the prior work using Bugs4Q, a widely used dataset of real-world bugs in quantum programs.<n>The experimental results showed that the Bugs4Q dropped from 62.2% on Qiskit v0.20.1 to 16.2% on v2.3.1, the latest version as of April 1, 2026.<n>While these findings are consistent with those of the prior work, we also observed differences. In particular, most reproduction failures in Bugs4Q cannot be resolved merely by adjusting dependency versions.
arXiv Detail & Related papers (2026-06-25T15:01:43Z)
Precise Debugging Benchmark: Is Your Model Debugging or Regenerating? [31.082688278576356]
We introduce the Precise Benchmark framework, which automatically converts any coding dataset into a benchmark with precision-aware evaluation.<n>We define two novel metrics, edit-level precision and bug-level recall, which measures how many necessary edits are made and how many bugs are resolved.<n>Experiments show frontier models, such as GPT-5.1-Codex and DeepSeek-V3.2-Thinking, achieve unit-test pass rates above 76% but exhibit precision below 45%.
arXiv Detail & Related papers (2026-04-19T09:08:23Z)
Are Benchmark Tests Strong Enough? Mutation-Guided Diagnosis and Augmentation of Regression Suites [49.16055123488827]
Insufficiently strong test suites can admit plausible yet semantically incorrect patches, inflating reported success rates.<n>We introduce STING, a framework for targeted test augmentation that uses semantically altered program variants as diagnostic stressors.<n>STING produces 1,014 validated tests spanning 211 instances and increases patch-region line and branch coverage by 10.8% and 9.5%.
arXiv Detail & Related papers (2026-04-02T01:13:40Z)
Outrunning LLM Cutoffs: A Live Kernel Crash Resolution Benchmark for All [57.23434868678603]
Live-kBench is an evaluation framework for self-evolving benchmarks that scrapes and evaluates agents on freshly discovered kernel bugs.<n> kEnv is an agent-agnostic crash-resolution environment for kernel compilation, execution, and feedback.<n>Using kEnv, we benchmark three state-of-the-art agents, showing that they resolve 74% of crashes on the first attempt.
arXiv Detail & Related papers (2026-02-02T19:06:15Z)
Specification-Guided Repair of Arithmetic Errors in Dafny Programs using LLMs [79.74676890436174]
We present an APR tool for Dafny that uses formal specifications as oracles for fault localization and repair.<n>We localize faults through a series of steps, which include using Hoare logic to determine the state of each statement within the program.<n>Our tool achieves 89.6% fault localization coverage and GPT-4o mini yields the highest repair success rate of 74.18%.
arXiv Detail & Related papers (2025-07-04T15:36:12Z)
Why you shouldn't fully trust ChatGPT: A synthesis of this AI tool's error rates across disciplines and the software engineering lifecycle [1.7912507269030578]
ChatGPT and other large language models (LLMs) are widely used across healthcare, business, economics, engineering, and software engineering (SE)<n>Despite their popularity, concerns persist about their reliability, especially their error rates across domains and the software development lifecycle (SDLC)<n>This study synthesizes and quantifies ChatGPT's reported error rates across major domains and SE tasks aligned with SDLC phases.
arXiv Detail & Related papers (2025-04-26T08:49:33Z)
Is Self-Repair a Silver Bullet for Code Generation? [68.02601393906083]
Large language models have shown remarkable aptitude in code generation, but still struggle to perform complex tasks. Self-repair -- in which the model debugs and repairs its own code -- has recently become a popular way to boost performance. We analyze Code Llama, GPT-3.5 and GPT-4's ability to perform self-repair on problems taken from HumanEval and APPS.
arXiv Detail & Related papers (2023-06-16T15:13:17Z)
QLoRA: Efficient Finetuning of Quantized LLMs [66.58009990713134]
We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU. QLoRA backpropagates through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters(LoRA) Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark.
arXiv Detail & Related papers (2023-05-23T17:50:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.