Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models
- URL: http://arxiv.org/abs/2406.14712v1
- Date: Thu, 20 Jun 2024 20:14:22 GMT
- Title: Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models
- Authors: Sanjay Vishwakarma, Francis Harkins, Siddharth Golecha, Vishal Sharathchandra Bajpe, Nicolas Dupuis, Luca Buratti, David Kremer, Ismael Faro, Ruchir Puri, Juan Cruz-Benito,
- Abstract summary: We introduce and use the Qiskit HumanEval dataset to benchmark the ability of Large Language Models to produce quantum code.
This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, and a difficulty scale to evaluate the correctness of the generated solutions.
- Score: 1.8213213818713139
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit - a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation.
Related papers
- Qiskit Code Assistant: Training LLMs for generating Quantum Computing Code [2.0108122340549985]
This paper focuses on training Code LLMs to specialize in the field of quantum computing.
A Code LLM specializing in quantum computing requires a foundational understanding of quantum computing and quantum information theory.
We discuss our work on training Code LLMs to produce high-quality quantum code using the Qiskit library.
arXiv Detail & Related papers (2024-05-29T20:21:00Z) - Quantum Computing Enhanced Service Ecosystem for Simulation in
Manufacturing [57.497038270123774]
We propose a framework for a quantum computing-enhanced service ecosystem for simulation in manufacturing.
We analyse two high-value use cases with the aim of a quantitative evaluation of these new computing paradigms for industrially-relevant settings.
arXiv Detail & Related papers (2024-01-19T11:04:14Z) - QKSAN: A Quantum Kernel Self-Attention Network [53.96779043113156]
A Quantum Kernel Self-Attention Mechanism (QKSAM) is introduced to combine the data representation merit of Quantum Kernel Methods (QKM) with the efficient information extraction capability of SAM.
A Quantum Kernel Self-Attention Network (QKSAN) framework is proposed based on QKSAM, which ingeniously incorporates the Deferred Measurement Principle (DMP) and conditional measurement techniques.
Four QKSAN sub-models are deployed on PennyLane and IBM Qiskit platforms to perform binary classification on MNIST and Fashion MNIST.
arXiv Detail & Related papers (2023-08-25T15:08:19Z) - Quafu-RL: The Cloud Quantum Computers based Quantum Reinforcement
Learning [0.0]
In this work, we take the first step towards executing benchmark quantum reinforcement problems on real devices equipped with at most 136 qubits on BAQIS Quafu quantum computing cloud.
The experimental results demonstrate that the Reinforcement Learning (RL) agents are capable of achieving goals that are slightly relaxed both during the training and inference stages.
arXiv Detail & Related papers (2023-05-29T09:13:50Z) - iQuantum: A Case for Modeling and Simulation of Quantum Computing
Environments [22.068803245816266]
iQuantum is a first-of-its-kind simulation toolkit that can model hybrid quantum-classical computing environments.
This paper presents the quantum computing system model, architectural design, proof-of-concept implementation, potential use cases, and future development of iQuantum.
arXiv Detail & Related papers (2023-03-28T04:51:32Z) - Preparing random state for quantum financing with quantum walks [1.2074552857379273]
We propose an efficient approach to load classical data into quantum states that can be executed by quantum computers or quantum simulators on classical hardware.
A practical example of implementing SSQW using Qiskit has been released as open-source software.
Showing its potential as a promising method for generating desired probability amplitude distributions highlights the potential application of SSQW in option pricing through quantum simulation.
arXiv Detail & Related papers (2023-02-24T08:01:35Z) - QSAN: A Near-term Achievable Quantum Self-Attention Network [73.15524926159702]
Self-Attention Mechanism (SAM) is good at capturing the internal connections of features.
A novel Quantum Self-Attention Network (QSAN) is proposed for image classification tasks on near-term quantum devices.
arXiv Detail & Related papers (2022-07-14T12:22:51Z) - Recent Advances for Quantum Neural Networks in Generative Learning [98.88205308106778]
Quantum generative learning models (QGLMs) may surpass their classical counterparts.
We review the current progress of QGLMs from the perspective of machine learning.
We discuss the potential applications of QGLMs in both conventional machine learning tasks and quantum physics.
arXiv Detail & Related papers (2022-06-07T07:32:57Z) - Generative Quantum Machine Learning [0.0]
The aim of this thesis is to develop new generative quantum machine learning algorithms.
We introduce a quantum generative adversarial network and a quantum Boltzmann machine implementation, both of which can be realized with parameterized quantum circuits.
arXiv Detail & Related papers (2021-11-24T19:00:21Z) - On exploring practical potentials of quantum auto-encoder with
advantages [92.19792304214303]
Quantum auto-encoder (QAE) is a powerful tool to relieve the curse of dimensionality encountered in quantum physics.
We prove that QAE can be used to efficiently calculate the eigenvalues and prepare the corresponding eigenvectors of a high-dimensional quantum state.
We devise three effective QAE-based learning protocols to solve the low-rank state fidelity estimation, the quantum Gibbs state preparation, and the quantum metrology tasks.
arXiv Detail & Related papers (2021-06-29T14:01:40Z) - Quantum Federated Learning with Quantum Data [87.49715898878858]
Quantum machine learning (QML) has emerged as a promising field that leans on the developments in quantum computing to explore large complex machine learning problems.
This paper proposes the first fully quantum federated learning framework that can operate over quantum data and, thus, share the learning of quantum circuit parameters in a decentralized manner.
arXiv Detail & Related papers (2021-05-30T12:19:27Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.