A Quantum Circuit-Based Compression Perspective for Parameter-Efficient Learning

• URL: http://arxiv.org/abs/2410.09846v1
• Date: Sun, 13 Oct 2024 14:09:29 GMT
• Title: A Quantum Circuit-Based Compression Perspective for Parameter-Efficient Learning
• Authors: Chen-Yu Liu, Chao-Han Huck Yang, Min-Hsiu Hsieh, Hsi-Sheng Goan,
• Abstract summary: We introduce Quantum s Adaptation (QPA) in the framework of quantum parameter generation. QPA integrates QNNs with a classical multi-layer perceptron mapping model to generate parameters for fine-tuning methods. Using Gemma-2 and GPT-2 as case studies, QPA demonstrates significant parameter reduction for parameter-efficient fine-tuning methods.
• Score: 19.178352290785153
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Quantum-centric supercomputing presents a compelling framework for large-scale hybrid quantum-classical tasks. Although quantum machine learning (QML) offers theoretical benefits in various applications, challenges such as large-size data encoding in the input stage and the reliance on quantum resources in the inference stage limit its practicality for tasks like fine-tuning large language models (LLMs). Quantum parameter generation, a novel approach of QML, addresses these limitations by using quantum neural networks (QNNs) to generate classical model weights (parameters) exclusively during training, thereby decoupling inference from quantum hardware. In this work, we introduce Quantum Parameter Adaptation (QPA) in the framework of quantum parameter generation, which integrates QNNs with a classical multi-layer perceptron mapping model to generate parameters for fine-tuning methods. Using Gemma-2 and GPT-2 as case studies, QPA demonstrates significant parameter reduction for parameter-efficient fine-tuning methods, such as Low-Rank Adaptation (LoRA), while maintaining comparable or improved performance in text generation tasks. Specifically, QPA reduces the number of parameters to $52.06\%$ of the original LoRA for GPT-2 with a slight performance gain of $0.75\%$, and to $16.84\%$ for Gemma-2, with a marginal performance improvement of $0.07\%$. These results highlight QPA's ability to achieve efficient parameter reduction without sacrificing performance in the quantum parameter generation framework. This work showcases the potential of quantum-enhanced parameter reduction, offering a scalable quantum-classical solution for fine-tuning LLMs while preserving the feasibility of inference on classical hardware.

Related papers

• Quantum Deep Equilibrium Models [1.5853439776721878]
  We present Quantum Deep Equilibrium Models (QDEQ), a training paradigm that learns parameters of a quantum machine learning model. We find that QDEQ is not only competitive with comparable existing baseline models, but also achieves higher performance than a network with 5 times more layers. This demonstrates that the QDEQ paradigm can be used to develop significantly more shallow quantum circuits for a given task.
  arXiv  Detail & Related papers  (2024-10-31T13:54:37Z)
• ML-QLS: Multilevel Quantum Layout Synthesis [6.706813469929441]
  We present ML-QLS, the first multilevel quantum layout tool with a scalable refinement operation integrated with novel cost functions and clustering strategies. Our experimental results demonstrate that ML-QLS can scale up to problems involving hundreds of qubits and achieve a remarkable 52% performance improvement over leading QLS tools for large circuits.
  arXiv  Detail & Related papers  (2024-05-28T17:10:20Z)
• A Quantum-Classical Collaborative Training Architecture Based on Quantum State Fidelity [50.387179833629254]
  We introduce a collaborative classical-quantum architecture called co-TenQu. Co-TenQu enhances a classical deep neural network by up to 41.72% in a fair setting. It outperforms other quantum-based methods by up to 1.9 times and achieves similar accuracy while utilizing 70.59% fewer qubits.
  arXiv  Detail & Related papers  (2024-02-23T14:09:41Z)
• QuantumSEA: In-Time Sparse Exploration for Noise Adaptive Quantum Circuits [82.50620782471485]
  QuantumSEA is an in-time sparse exploration for noise-adaptive quantum circuits. It aims to achieve two key objectives: (1) implicit circuits capacity during training and (2) noise robustness. Our method establishes state-of-the-art results with only half the number of quantum gates and 2x time saving of circuit executions.
  arXiv  Detail & Related papers  (2024-01-10T22:33:00Z)
• Pre-optimizing variational quantum eigensolvers with tensor networks [1.4512477254432858]
  We present and benchmark an approach where we find good starting parameters for parameterized quantum circuits by simulating VQE. We apply this approach to the 1D and 2D Fermi-Hubbard model with system sizes that use up to 32 qubits. In 2D, the parameters that VTNE finds have significantly lower energy than their starting configurations, and we show that starting VQE from these parameters requires non-trivially fewer operations to come down to a given energy.
  arXiv  Detail & Related papers  (2023-10-19T17:57:58Z)
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models [57.27101446992148]
  Large language models (LLMs) have revolutionized natural language processing tasks. Recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM. We introduce an Omnidirectionally calibrated Quantization technique for LLMs, which achieves good performance in diverse quantization settings.
  arXiv  Detail & Related papers  (2023-08-25T02:28:35Z)
• Restricting to the chip architecture maintains the quantum neural network accuracy [0.0]
  variational quantum algorithms (VQAs) are a prominent strategy for constructing quantum machine learning models. This article aims to address several critical questions, such as determining the optimal gate sequence, efficient parameter optimization strategies, and understanding the influence of quantum chip architectures on the final results.
  arXiv  Detail & Related papers  (2022-12-29T18:57:05Z)
• Symmetric Pruning in Quantum Neural Networks [111.438286016951]
  Quantum neural networks (QNNs) exert the power of modern quantum machines. QNNs with handcraft symmetric ansatzes generally experience better trainability than those with asymmetric ansatzes. We propose the effective quantum neural tangent kernel (EQNTK) to quantify the convergence of QNNs towards the global optima.
  arXiv  Detail & Related papers  (2022-08-30T08:17:55Z)
• Variational Quantum Reinforcement Learning via Evolutionary Optimization [0.0]
  We present two frameworks of deep quantum RL tasks using a gradient-free evolution optimization. We propose a hybrid framework where the quantum RL agents are equipped with hybrid tensor network-variational quantum circuit (TN-VQC) architecture. This allows us to perform quantum RL on the MiniGrid environment with 147-dimensional inputs.
  arXiv  Detail & Related papers  (2021-09-01T16:36:04Z)
• FLIP: A flexible initializer for arbitrarily-sized parametrized quantum circuits [105.54048699217668]
  We propose a FLexible Initializer for arbitrarily-sized Parametrized quantum circuits. FLIP can be applied to any family of PQCs, and instead of relying on a generic set of initial parameters, it is tailored to learn the structure of successful parameters. We illustrate the advantage of using FLIP in three scenarios: a family of problems with proven barren plateaus, PQC training to solve max-cut problem instances, and PQC training for finding the ground state energies of 1D Fermi-Hubbard models.
  arXiv  Detail & Related papers  (2021-03-15T17:38:33Z)
• Quantum circuit architecture search for variational quantum algorithms [88.71725630554758]
  We propose a resource and runtime efficient scheme termed quantum architecture search (QAS) QAS automatically seeks a near-optimal ansatz to balance benefits and side-effects brought by adding more noisy quantum gates. We implement QAS on both the numerical simulator and real quantum hardware, via the IBM cloud, to accomplish data classification and quantum chemistry tasks.
  arXiv  Detail & Related papers  (2020-10-20T12:06:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.