Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models

• URL: http://arxiv.org/abs/2406.14712v1
• Date: Thu, 20 Jun 2024 20:14:22 GMT
• Title: Qiskit HumanEval: An Evaluation Benchmark For Quantum Code Generative Models
• Authors: Sanjay Vishwakarma, Francis Harkins, Siddharth Golecha, Vishal Sharathchandra Bajpe, Nicolas Dupuis, Luca Buratti, David Kremer, Ismael Faro, Ruchir Puri, Juan Cruz-Benito,
• Abstract summary: We introduce and use the Qiskit HumanEval dataset to benchmark the ability of Large Language Models to produce quantum code. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, and a difficulty scale to evaluate the correctness of the generated solutions.
• Score: 1.8213213818713139
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quantum programs are typically developed using quantum Software Development Kits (SDKs). The rapid advancement of quantum computing necessitates new tools to streamline this development process, and one such tool could be Generative Artificial intelligence (GenAI). In this study, we introduce and use the Qiskit HumanEval dataset, a hand-curated collection of tasks designed to benchmark the ability of Large Language Models (LLMs) to produce quantum code using Qiskit - a quantum SDK. This dataset consists of more than 100 quantum computing tasks, each accompanied by a prompt, a canonical solution, a comprehensive test case, and a difficulty scale to evaluate the correctness of the generated solutions. We systematically assess the performance of a set of LLMs against the Qiskit HumanEval dataset's tasks and focus on the models ability in producing executable quantum code. Our findings not only demonstrate the feasibility of using LLMs for generating quantum code but also establish a new benchmark for ongoing advancements in the field and encourage further exploration and development of GenAI-driven tools for quantum code generation.

Related papers

• QCircuitNet: A Large-Scale Hierarchical Dataset for Quantum Algorithm Design [17.747641494506087]
  We introduce QCircuitNet, the first benchmark and test dataset designed to evaluate AI's capability in designing and implementing quantum algorithms. Unlike using AI for writing traditional codes, this task is fundamentally different and significantly more complicated due to highly flexible design space and intricate manipulation of qubits.
  arXiv  Detail & Related papers  (2024-10-10T14:24:30Z)
• LatentQGAN: A Hybrid QGAN with Classical Convolutional Autoencoder [7.945302052915863]
  A potential application of quantum machine learning is to harness the power of quantum computers for generating classical data. We propose LatentQGAN, a novel quantum model that uses a hybrid quantum-classical GAN coupled with an autoencoder.
  arXiv  Detail & Related papers  (2024-09-22T23:18:06Z)
• The curse of random quantum data [62.24825255497622]
  We quantify the performances of quantum machine learning in the landscape of quantum data. We find that the training efficiency and generalization capabilities in quantum machine learning will be exponentially suppressed with the increase in qubits. Our findings apply to both the quantum kernel method and the large-width limit of quantum neural networks.
  arXiv  Detail & Related papers  (2024-08-19T12:18:07Z)
• Qiskit Code Assistant: Training LLMs for generating Quantum Computing Code [2.0108122340549985]
  This paper focuses on training Code LLMs to specialize in the field of quantum computing. A Code LLM specializing in quantum computing requires a foundational understanding of quantum computing and quantum information theory. We discuss our work on training Code LLMs to produce high-quality quantum code using the Qiskit library.
  arXiv  Detail & Related papers  (2024-05-29T20:21:00Z)
• Quantum Computing Enhanced Service Ecosystem for Simulation in Manufacturing [56.61654656648898]
  We propose a framework for a quantum computing-enhanced service ecosystem for simulation in manufacturing. We analyse two high-value use cases with the aim of a quantitative evaluation of these new computing paradigms for industrially-relevant settings.
  arXiv  Detail & Related papers  (2024-01-19T11:04:14Z)
• QKSAN: A Quantum Kernel Self-Attention Network [53.96779043113156]
  A Quantum Kernel Self-Attention Mechanism (QKSAM) is introduced to combine the data representation merit of Quantum Kernel Methods (QKM) with the efficient information extraction capability of SAM. A Quantum Kernel Self-Attention Network (QKSAN) framework is proposed based on QKSAM, which ingeniously incorporates the Deferred Measurement Principle (DMP) and conditional measurement techniques. Four QKSAN sub-models are deployed on PennyLane and IBM Qiskit platforms to perform binary classification on MNIST and Fashion MNIST.
  arXiv  Detail & Related papers  (2023-08-25T15:08:19Z)
• iQuantum: A Case for Modeling and Simulation of Quantum Computing Environments [22.068803245816266]
  iQuantum is a first-of-its-kind simulation toolkit that can model hybrid quantum-classical computing environments. This paper presents the quantum computing system model, architectural design, proof-of-concept implementation, potential use cases, and future development of iQuantum.
  arXiv  Detail & Related papers  (2023-03-28T04:51:32Z)
• Preparing random state for quantum financing with quantum walks [1.2074552857379273]
  We propose an efficient approach to load classical data into quantum states that can be executed by quantum computers or quantum simulators on classical hardware. A practical example of implementing SSQW using Qiskit has been released as open-source software. Showing its potential as a promising method for generating desired probability amplitude distributions highlights the potential application of SSQW in option pricing through quantum simulation.
  arXiv  Detail & Related papers  (2023-02-24T08:01:35Z)
• Recent Advances for Quantum Neural Networks in Generative Learning [98.88205308106778]
  Quantum generative learning models (QGLMs) may surpass their classical counterparts. We review the current progress of QGLMs from the perspective of machine learning. We discuss the potential applications of QGLMs in both conventional machine learning tasks and quantum physics.
  arXiv  Detail & Related papers  (2022-06-07T07:32:57Z)
• On exploring the potential of quantum auto-encoder for learning quantum systems [60.909817434753315]
  We devise three effective QAE-based learning protocols to address three classically computational hard learning problems. Our work sheds new light on developing advanced quantum learning algorithms to accomplish hard quantum physics and quantum information processing tasks.
  arXiv  Detail & Related papers  (2021-06-29T14:01:40Z)
• Quantum Federated Learning with Quantum Data [87.49715898878858]
  Quantum machine learning (QML) has emerged as a promising field that leans on the developments in quantum computing to explore large complex machine learning problems. This paper proposes the first fully quantum federated learning framework that can operate over quantum data and, thus, share the learning of quantum circuit parameters in a decentralized manner.
  arXiv  Detail & Related papers  (2021-05-30T12:19:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.