QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges

• URL: http://arxiv.org/abs/2506.20008v2
• Date: Fri, 29 Aug 2025 07:06:23 GMT
• Title: QHackBench: Benchmarking Large Language Models for Quantum Code Generation Using PennyLane Hackathon Challenges
• Abstract summary: This paper benchmarks Large Language Models for PennyLane-based quantum code generation using real-world challenges from the Quantum Hackathon (QHack)<n>We introduce QHackBench, a novel benchmark dataset derived from QHack competitions, and evaluate model performance under vanilla prompting and Retrieval-Augmented Generation (RAG)<n>Results indicate that RAG-enhanced models, supplemented with an augmented PennyLane dataset, approximately generate similar results as the standard prompting, particularly in complex quantum algorithms.
• Score: 4.097173414006477
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in Large Language Models (LLMs) have demonstrated strong potential in code generation, yet their effectiveness in quantum computing remains underexplored. This paper benchmarks LLMs for PennyLane-based quantum code generation using real-world challenges from the Quantum Hackathon (QHack). We introduce QHackBench, a novel benchmark dataset derived from QHack competitions, and evaluate model performance under vanilla prompting and Retrieval-Augmented Generation (RAG). Our structured evaluation framework assesses functional correctness, syntactic validity, and execution success across varying challenge difficulties. Results indicate that RAG-enhanced models, supplemented with an augmented PennyLane dataset, approximately generate similar results as the standard prompting, particularly in complex quantum algorithms. Additionally, we introduce a multi-agent evaluation pipeline that iteratively refines incorrect solutions, further enhancing execution success rates. To foster further research, we commit to publicly releasing QHackBench, along with our evaluation framework and experimental results, enabling continued advancements in AI-assisted quantum programming.

Related papers

• Hybrid Quantum-Classical Neural Networks for Few-Shot Credit Risk Assessment [52.05742536403784]
  This work tackles the challenge of few-shot credit risk assessment.<n>We design and implement a novel hybrid quantum-classical workflow.<n>A Quantum Neural Network (QNN) was trained via the parameter-shift rule.<n>On a real-world credit dataset of 279 samples, our QNN achieved a robust average AUC of 0.852 +/- 0.027 in simulations and yielded an impressive AUC of 0.88 in the hardware experiment.
  arXiv  Detail & Related papers  (2025-09-17T08:36:05Z)
• LLM-Guided Ansätze Design for Quantum Circuit Born Machines in Financial Generative Modeling [0.9176056742068813]
  Quantum generative modeling using quantum circuit Born machines (QCBMs) shows promising potential for practical quantum advantage.<n>We introduce a prompt-based framework that leverages large language models (LLMs) to generate hardware-aware QCBM architectures.<n>We show that the LLM-generated ans"atze are significantly shallower and achieve superior generative performance compared to the standard baseline when executed on real IBM quantum hardware using 12 qubits.
  arXiv  Detail & Related papers  (2025-09-10T08:23:58Z)
• Resource-Efficient Hadamard Test Circuits for Nonlinear Dynamics on a Trapped-Ion Quantum Computer [1.2063443893298391]
  We propose a low-depth implementation of a class of Hadamard test circuits.<n>We develop a parameterized quantum ansatz specifically tailored for variational algorithms.<n>Our findings demonstrate a significant reduction in single- and two-qubit gate counts.
  arXiv  Detail & Related papers  (2025-07-25T13:16:54Z)
• Awesome Quantum Computing Experiments: Benchmarking Experimental Progress Towards Fault-Tolerant Quantum Computation [0.0]
  This work reviews and benchmarks experimental advancements towards fault-tolerant quantum computation (FTQC)<n>We analyze key physical metrics like coherence times, entanglement error, and system size (qubit count)<n>At the logical level, we survey the implementation landscape of QEC codes.
  arXiv  Detail & Related papers  (2025-07-04T15:56:28Z)
• Segmentation-Based Regression for Quantum Neural Networks [0.0]
  Recent advances in quantum hardware motivate the development of algorithmic frameworks that integrate quantum sampling with classical inference.<n>This work introduces a segmentation-based regression method tailored to quantum neural networks (QNNs)<n>By casting the regression task as a constrained problem over a structured digit lattice, the method replaces continuous inference with interpretable and tractable updates.
  arXiv  Detail & Related papers  (2025-06-27T20:11:43Z)
• VQC-MLPNet: An Unconventional Hybrid Quantum-Classical Architecture for Scalable and Robust Quantum Machine Learning [60.996803677584424]
  Variational Quantum Circuits (VQCs) offer a novel pathway for quantum machine learning.<n>Their practical application is hindered by inherent limitations such as constrained linear expressivity, optimization challenges, and acute sensitivity to quantum hardware noise.<n>This work introduces VQC-MLPNet, a scalable and robust hybrid quantum-classical architecture designed to overcome these obstacles.
  arXiv  Detail & Related papers  (2025-06-12T01:38:15Z)
• An Efficient Quantum Classifier Based on Hamiltonian Representations [50.467930253994155]
  Quantum machine learning (QML) is a discipline that seeks to transfer the advantages of quantum computing to data-driven tasks.<n>We propose an efficient approach that circumvents the costs associated with data encoding by mapping inputs to a finite set of Pauli strings.<n>We evaluate our approach on text and image classification tasks, against well-established classical and quantum models.
  arXiv  Detail & Related papers  (2025-04-13T11:49:53Z)
• Quantum Bayesian Networks for Machine Learning in Oil-Spill Detection [3.9554540293311864]
  Quantum Machine Learning has shown promise in diverse applications such as environmental monitoring, healthcare diagnostics, and financial modeling.<n>One critical challenge is handling imbalanced datasets, where rare events are often misclassified due to skewed data distributions.<n>This paper introduces a Bayesian approach utilizing QBNs to classify satellite-derived imbalanced datasets, distinguishing oil-spill'' from non-spill'' regions.
  arXiv  Detail & Related papers  (2024-12-24T15:44:26Z)
• QCircuitBench: A Large-Scale Dataset for Benchmarking Quantum Algorithm Design [63.02824918725805]
  Quantum computing is recognized for the significant speedup it offers over classical computing through quantum algorithms.<n>QCircuitBench is the first benchmark dataset designed to evaluate AI's capability in designing and implementing quantum algorithms.
  arXiv  Detail & Related papers  (2024-10-10T14:24:30Z)
• AdaLog: Post-Training Quantization for Vision Transformers with Adaptive Logarithm Quantizer [54.713778961605115]
  Vision Transformer (ViT) has become one of the most prevailing fundamental backbone networks in the computer vision community. We propose a novel non-uniform quantizer, dubbed the Adaptive Logarithm AdaLog (AdaLog) quantizer.
  arXiv  Detail & Related papers  (2024-07-17T18:38:48Z)
• Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models [1.8213213818713139]
  We introduce and use the Qiskit HumanEval dataset to benchmark the ability of Large Language Models to produce quantum code. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, and a difficulty scale to evaluate the correctness of the generated solutions.
  arXiv  Detail & Related papers  (2024-06-20T20:14:22Z)
• Stressing Out Modern Quantum Hardware: Performance Evaluation and Execution Insights [2.2091590689610823]
  Stress testing is a technique used to evaluate a system by giving it a computational load beyond its specified thresholds. We conduct a qualitative and quantitative evaluation of the Quantinuum H1 ion trap device using a stress test based protocol.
  arXiv  Detail & Related papers  (2024-01-24T20:22:34Z)
• Quantum Imitation Learning [74.15588381240795]
  We propose quantum imitation learning (QIL) with a hope to utilize quantum advantage to speed up IL. We develop two QIL algorithms, quantum behavioural cloning (Q-BC) and quantum generative adversarial imitation learning (Q-GAIL) Experiment results demonstrate that both Q-BC and Q-GAIL can achieve comparable performance compared to classical counterparts.
  arXiv  Detail & Related papers  (2023-04-04T12:47:35Z)
• Optimization Applications as Quantum Performance Benchmarks [0.0]
  Combinatorial optimization is anticipated to be one of the primary use cases for quantum computation in the coming years. Inspired by existing methods to characterize classical optimization algorithms, we analyze the solution quality obtained by solving Max-Cut problems. This is used to guide the development of an advanced benchmarking framework for quantum computers.
  arXiv  Detail & Related papers  (2023-02-05T01:56:06Z)
• QuanGCN: Noise-Adaptive Training for Robust Quantum Graph Convolutional Networks [124.7972093110732]
  We propose quantum graph convolutional networks (QuanGCN), which learns the local message passing among nodes with the sequence of crossing-gate quantum operations. To mitigate the inherent noises from modern quantum devices, we apply sparse constraint to sparsify the nodes' connections. Our QuanGCN is functionally comparable or even superior than the classical algorithms on several benchmark graph datasets.
  arXiv  Detail & Related papers  (2022-11-09T21:43:16Z)
• Quantum circuit architecture search for variational quantum algorithms [88.71725630554758]
  We propose a resource and runtime efficient scheme termed quantum architecture search (QAS) QAS automatically seeks a near-optimal ansatz to balance benefits and side-effects brought by adding more noisy quantum gates. We implement QAS on both the numerical simulator and real quantum hardware, via the IBM cloud, to accomplish data classification and quantum chemistry tasks.
  arXiv  Detail & Related papers  (2020-10-20T12:06:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.