QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation

• URL: http://arxiv.org/abs/2604.08570v1
• Date: Wed, 25 Mar 2026 20:51:21 GMT
• Title: QuanBench+: A Unified Multi-Framework Benchmark for LLM-Based Quantum Code Generation
• Abstract summary: We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq.<n>We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs.<n>Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane.
• Score: 39.32132630606808
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) are increasingly used for code generation, yet quantum code generation is still evaluated mostly within single frameworks, making it difficult to separate quantum reasoning from framework familiarity. We introduce QuanBench+, a unified benchmark spanning Qiskit, PennyLane, and Cirq, with 42 aligned tasks covering quantum algorithms, gate decomposition, and state preparation. We evaluate models with executable functional tests, report Pass@1 and Pass@5, and use KL-divergence-based acceptance for probabilistic outputs. We additionally study Pass@1 after feedback-based repair, where a model may revise code after a runtime error or wrong answer. Across frameworks, the strongest one-shot scores reach 59.5% in Qiskit, 54.8% in Cirq, and 42.9% in PennyLane; with feedback-based repair, the best scores rise to 83.3%, 76.2%, and 66.7%, respectively. These results show clear progress, but also that reliable multi-framework quantum code generation remains unsolved and still depends strongly on framework-specific knowledge.

Related papers

• Benchmarking Agents for Proving Theorems in Quantum Algorithms and Quantum Information [13.46344583931603]
  We introduce Lean-QuantumAlg-Bench and Lean-QIT-Bench, two Lean 4 benchmarks containing 36 and 40 theorem-completion tasks for quantum algorithms and quantum information theory, respectively.<n>We evaluate four models-GPT-5.5, Kimi K3, DeepSeek V4-Pro, and MiniMax M3-within a common theorem-proving framework under two settings.<n>The highest difficulty-weighted scores are 60.4 out of 100 on the quantum-algorithm benchmark and 59.6 out of 100 on the quantum-information benchmark.
  arXiv  Detail & Related papers  (2026-07-23T17:19:21Z)
• QBugLM: An Agentic Benchmarking Framework for LLM-based Quantum Software Debugging [0.8609132348927196]
  Large language models (LLMs) have shown strong performance on classical software engineering tasks.<n>We propose QBugLM, a multi-agent framework that automates the quantum software debug pipeline.
  arXiv  Detail & Related papers  (2026-06-05T14:34:46Z)
• Qiskit QuantumKatas: Adapting Microsoft's Quantum Computing exercises for LLM evaluation [1.2891210250935148]
  We package Microsoft's QuantumKatas quantum computing curriculum with an evaluation framework for systematic LLM assessment.<n>The resulting benchmark comprises 350 tasks across 26 categories, spanning fundamental gates through advanced algorithms.<n>Each task includes a natural language prompt, canonical solution, and deterministic test verification via classical circuit simulation.
  arXiv  Detail & Related papers  (2026-05-26T16:02:08Z)
• Quantum-Safe Code Auditing: LLM-Assisted Static Analysis and Quantum-Aware Risk Scoring for Post-Quantum Cryptography Migration [0.0]
  Quantum-Safe Code is a quantum-aware static analysis framework.<n>It uses 15 classes of quantum-vulnerable primitives and a Variational Bouncysolver (VQE) model.<n>On a sample of 602 labelled instances, we achieve 71.98% precision, 100% recall, and an F1 score of 83.71%.
  arXiv  Detail & Related papers  (2026-04-01T07:10:17Z)
• Omanic: Towards Step-wise Evaluation of Multi-hop Reasoning in Large Language Models [60.418191092851636]
  Omanic is an open-domain multi-hop QA resource that provides decomposed sub-questions and intermediate answers as structural annotations for analyzing reasoning processes.<n>It contains 10,296 machine-generated training examples (Omanic Synth) and 967 expert-reviewed human-annotated evaluation examples (OmanicBench)
  arXiv  Detail & Related papers  (2026-03-17T15:23:37Z)
• QuanBench: Benchmarking Quantum Code Generation with Large Language Models [7.807551490308163]
  Large language models (LLMs) have demonstrated good performance in general code generation.<n>This paper presents QuanBench, a benchmark for evaluating LLMs on quantum code generation.
  arXiv  Detail & Related papers  (2025-10-19T10:08:36Z)
• Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning [53.45095336430027]
  We develop a unified framework that combines implicit retrieval and structured collaboration.<n>On Humanity's Last Exam (HLE) Bio/Chem Gold, our framework achieves 48.3% accuracy.<n>Results on SuperGPQA and TRQA confirm robustness across domains.
  arXiv  Detail & Related papers  (2025-09-25T14:05:55Z)
• QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges [4.097173414006477]
  This paper benchmarks Large Language Models for PennyLane-based quantum code generation using real-world challenges from the Quantum Hackathon (QHack)<n>We introduce QHackBench, a novel benchmark dataset derived from QHack competitions, and evaluate model performance under vanilla prompting and Retrieval-Augmented Generation (RAG)<n>Results indicate that RAG-enhanced models, supplemented with an augmented PennyLane dataset, approximately generate similar results as the standard prompting, particularly in complex quantum algorithms.
  arXiv  Detail & Related papers  (2025-06-24T20:54:56Z)
• EquiBench: Benchmarking Large Language Models' Reasoning about Program Semantics via Equivalence Checking [58.15568681219339]
  We introduce EquiBench, a new benchmark for evaluating large language models (LLMs)<n>This task directly tests a model's ability to reason about program semantics.<n>We evaluate 19 state-of-the-art LLMs and find that in the most challenging categories, the best accuracies are 63.8% and 76.2%, only modestly above the 50% random baseline.
  arXiv  Detail & Related papers  (2025-02-18T02:54:25Z)
• ProofAug: Efficient Neural Theorem Proving via Fine-grained Proof Structure Analysis [50.020850767257095]
  We propose ProofAug, a procedure that equips LLMs with automation methods at various granularities.<n>Our method is validated on the miniF2F benchmark using the open-source deep-math-7b-base model and the Isabelle proof assistant.<n>We also implement a Lean 4 version of ProofAug that can improve the pass@1 performance of Kimina-Prover-seek-Distill-1.5B from 44.3% to 50.4%.
  arXiv  Detail & Related papers  (2025-01-30T12:37:06Z)
• "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization [67.3213104337679]
  Quantization is a powerful tool for accelerating large language model (LLM) inference, but the accuracy-performance trade-offs across different formats remain unclear.<n>We conduct the most comprehensive empirical study to date, evaluating FP8, INT8, and INT4 quantization across academic benchmarks and real-world tasks.
  arXiv  Detail & Related papers  (2024-11-04T18:21:59Z)
• Analyzing Quantum Programs with LintQ: A Static Analysis Framework for Qiskit [21.351834312054844]
  This paper presents LintQ, a comprehensive static analysis framework for detecting bugs in quantum programs. Our approach is enabled by a set of abstractions designed to reason about common concepts in quantum computing without referring to the underlying quantum computing platform. We apply the approach to a newly collected dataset of 7,568 real-world Qiskit-based quantum programs, showing that LintQ effectively identifies various programming problems.
  arXiv  Detail & Related papers  (2023-10-01T16:36:09Z)
• PRover: Proof Generation for Interpretable Reasoning over Rules [81.40404921232192]
  We propose a transformer-based model that answers binary questions over rule-bases and generates the corresponding proofs. Our model learns to predict nodes and edges corresponding to proof graphs in an efficient constrained training paradigm. We conduct experiments on synthetic, hand-authored, and human-paraphrased rule-bases to show promising results for QA and proof generation.
  arXiv  Detail & Related papers  (2020-10-06T15:47:53Z)
• QUANTIFY: A framework for resource analysis and design verification of quantum circuits [69.43216268165402]
  QUANTIFY is an open-source framework for the quantitative analysis of quantum circuits. It is based on Google Cirq and is developed with Clifford+T circuits in mind. For benchmarking purposes QUANTIFY includes quantum memory and quantum arithmetic circuits.
  arXiv  Detail & Related papers  (2020-07-21T15:36:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.