PPO-Q: Proximal Policy Optimization with Parametrized Quantum Policies or Values

• URL: http://arxiv.org/abs/2501.07085v1
• Date: Mon, 13 Jan 2025 06:40:40 GMT
• Title: PPO-Q: Proximal Policy Optimization with Parametrized Quantum Policies or Values
• Authors: Yu-Xin Jin, Zi-Wei Wang, Hong-Ze Xu, Wei-Feng Zhuang, Meng-Jun Hu, Dong E. Liu,
• Abstract summary: PPO-Q integrates hybrid quantum-classical networks into the actor or critic part of the proximal policy optimization (PPO) algorithm.<n>The PPO-Q achieves state-of-the-art performance in a range of complex environments with significantly reduced training parameters.
• Score: 5.260281988042923
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quantum machine learning (QML), which combines quantum computing with machine learning, is widely believed to hold the potential to outperform traditional machine learning in the era of noisy intermediate-scale quantum (NISQ). As one of the most important types of QML, quantum reinforcement learning (QRL) with parameterized quantum circuits as agents has received extensive attention in the past few years. Various algorithms and techniques have been introduced, demonstrating the effectiveness of QRL in solving some popular benchmark environments such as CartPole, FrozenLake, and MountainCar. However, tackling more complex environments with continuous action spaces and high-dimensional state spaces remains challenging within the existing QRL framework. Here we present PPO-Q, which, by integrating hybrid quantum-classical networks into the actor or critic part of the proximal policy optimization (PPO) algorithm, achieves state-of-the-art performance in a range of complex environments with significantly reduced training parameters. The hybrid quantum-classical networks in the PPO-Q incorporate two additional traditional neural networks to aid the parameterized quantum circuits in managing high-dimensional state encoding and action selection. When evaluated on 8 diverse environments, including four with continuous action space, the PPO-Q achieved comparable performance with the PPO algorithm but with significantly reduced training parameters. Especially, we accomplished the BipedalWalker environment, with a high-dimensional state and continuous action space simultaneously, which has not previously been reported in the QRL. More importantly, the PPO-Q is very friendly to the current NISQ hardware. We successfully trained two representative environments on the real superconducting quantum devices via the Quafu quantum cloud service.

Related papers

• Extending Quantum Perceptrons: Rydberg Devices, Multi-Class Classification, and Error Tolerance [67.77677387243135]
  Quantum Neuromorphic Computing (QNC) merges quantum computation with neural computation to create scalable, noise-resilient algorithms for quantum machine learning (QML) At the core of QNC is the quantum perceptron (QP), which leverages the analog dynamics of interacting qubits to enable universal quantum computation.
  arXiv  Detail & Related papers  (2024-11-13T23:56:20Z)
• KANQAS: Kolmogorov-Arnold Network for Quantum Architecture Search [0.0]
  We use the Kolmogorov-Arnold Network (KAN) in the Quantum Search (QAS) algorithm, analyzing their efficiency in the task of quantum state preparation and quantum chemistry.<n>In quantum state preparation, our results show that in a noiseless scenario, the probability of success is 2 to 5 times higher than robustnesss.<n>In tackling quantum chemistry problems, we enhance the recently proposed QAS algorithm by integrating curriculum reinforcement learning with a KAN structure.
  arXiv  Detail & Related papers  (2024-06-25T15:17:01Z)
• A Quantum-Classical Collaborative Training Architecture Based on Quantum State Fidelity [50.387179833629254]
  We introduce a collaborative classical-quantum architecture called co-TenQu. Co-TenQu enhances a classical deep neural network by up to 41.72% in a fair setting. It outperforms other quantum-based methods by up to 1.9 times and achieves similar accuracy while utilizing 70.59% fewer qubits.
  arXiv  Detail & Related papers  (2024-02-23T14:09:41Z)
• A joint optimization approach of parameterized quantum circuits with a tensor network [0.0]
  Current intermediate-scale quantum (NISQ) devices remain limited in their capabilities. We propose the use of parameterized Networks (TNs) to attempt an improved performance of the Variational Quantum Eigensolver (VQE) algorithm.
  arXiv  Detail & Related papers  (2024-02-19T12:53:52Z)
• QuantumSEA: In-Time Sparse Exploration for Noise Adaptive Quantum Circuits [82.50620782471485]
  QuantumSEA is an in-time sparse exploration for noise-adaptive quantum circuits. It aims to achieve two key objectives: (1) implicit circuits capacity during training and (2) noise robustness. Our method establishes state-of-the-art results with only half the number of quantum gates and 2x time saving of circuit executions.
  arXiv  Detail & Related papers  (2024-01-10T22:33:00Z)
• Non-asymptotic Approximation Error Bounds of Parameterized Quantum Circuits [16.460585387762478]
  ized quantum circuits (PQCs) have emerged as a promising approach for quantum neural networks. This paper investigates the expressivity of PQCs for approximating general function classes. We establish the first non-asymptotic approximation error bounds for these functions in terms of the number of qubits, quantum circuit depth, and number of trainable parameters.
  arXiv  Detail & Related papers  (2023-10-11T14:29:11Z)
• A self-consistent field approach for the variational quantum eigensolver: orbital optimization goes adaptive [52.77024349608834]
  We present a self consistent field approach (SCF) within the Adaptive Derivative-Assembled Problem-Assembled Ansatz Variational Eigensolver (ADAPTVQE) This framework is used for efficient quantum simulations of chemical systems on nearterm quantum computers.
  arXiv  Detail & Related papers  (2022-12-21T23:15:17Z)
• Evaluation of Parameterized Quantum Circuits with Cross-Resonance Pulse-Driven Entanglers [0.27998963147546146]
  Variational Quantum Algorithms (VQAs) have emerged as a powerful class of algorithms that is highly suitable for noisy quantum devices. Previous works have shown that choosing an effective parameterized quantum circuit (PQC) or ansatz for VQAs is crucial to their overall performance. In this paper, we utilize pulse-level access to quantum machines and our understanding of their two-qubit interactions to optimize the design of two-qubit entanglers.
  arXiv  Detail & Related papers  (2022-11-01T09:46:34Z)
• Synergy Between Quantum Circuits and Tensor Networks: Short-cutting the Race to Practical Quantum Advantage [43.3054117987806]
  We introduce a scalable procedure for harnessing classical computing resources to provide pre-optimized initializations for quantum circuits. We show this method significantly improves the trainability and performance of PQCs on a variety of problems. By demonstrating a means of boosting limited quantum resources using classical computers, our approach illustrates the promise of this synergy between quantum and quantum-inspired models in quantum computing.
  arXiv  Detail & Related papers  (2022-08-29T15:24:03Z)
• Quantum agents in the Gym: a variational quantum algorithm for deep Q-learning [0.0]
  We introduce a training method for parametrized quantum circuits (PQCs) that can be used to solve RL tasks for discrete and continuous state spaces. We investigate which architectural choices for quantum Q-learning agents are most important for successfully solving certain types of environments.
  arXiv  Detail & Related papers  (2021-03-28T08:57:22Z)
• FLIP: A flexible initializer for arbitrarily-sized parametrized quantum circuits [105.54048699217668]
  We propose a FLexible Initializer for arbitrarily-sized Parametrized quantum circuits. FLIP can be applied to any family of PQCs, and instead of relying on a generic set of initial parameters, it is tailored to learn the structure of successful parameters. We illustrate the advantage of using FLIP in three scenarios: a family of problems with proven barren plateaus, PQC training to solve max-cut problem instances, and PQC training for finding the ground state energies of 1D Fermi-Hubbard models.
  arXiv  Detail & Related papers  (2021-03-15T17:38:33Z)
• Quantum circuit architecture search for variational quantum algorithms [88.71725630554758]
  We propose a resource and runtime efficient scheme termed quantum architecture search (QAS) QAS automatically seeks a near-optimal ansatz to balance benefits and side-effects brought by adding more noisy quantum gates. We implement QAS on both the numerical simulator and real quantum hardware, via the IBM cloud, to accomplish data classification and quantum chemistry tasks.
  arXiv  Detail & Related papers  (2020-10-20T12:06:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.