Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention

• URL: http://arxiv.org/abs/2403.14753v1
• Date: Thu, 21 Mar 2024 18:00:04 GMT
• Title: Learning with SASQuaTCh: a Novel Variational Quantum Transformer Architecture with Kernel-Based Self-Attention
• Authors: Ethan N. Evans, Matthew Cook, Zachary P. Bradshaw, Margarite L. LaBorde,
• Abstract summary: We show that quantum circuits can efficiently express a self-attention mechanism through the perspective of kernel-based operator learning. In this work, we are able to represent deep layers of a vision transformer network using simple gate operations and a set of multi-dimensional quantum Fourier transforms. We analyze our novel variational quantum circuit, which we call Self-Attention Sequential Quantum Transformer Channel (SASTQuaCh), and demonstrate its utility on simplified classification problems.
• Score: 0.464982780843177
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The widely popular transformer network popularized by the generative pre-trained transformer (GPT) has a large field of applicability, including predicting text and images, classification, and even predicting solutions to the dynamics of physical systems. In the latter context, the continuous analog of the self-attention mechanism at the heart of transformer networks has been applied to learning the solutions of partial differential equations and reveals a convolution kernel nature that can be exploited by the Fourier transform. It is well known that many quantum algorithms that have provably demonstrated a speedup over classical algorithms utilize the quantum Fourier transform. In this work, we explore quantum circuits that can efficiently express a self-attention mechanism through the perspective of kernel-based operator learning. In this perspective, we are able to represent deep layers of a vision transformer network using simple gate operations and a set of multi-dimensional quantum Fourier transforms. We analyze the computational and parameter complexity of our novel variational quantum circuit, which we call Self-Attention Sequential Quantum Transformer Channel (SASQuaTCh), and demonstrate its utility on simplified classification problems.

Related papers

• Quantum parallel information exchange (QPIE) hybrid network with transfer learning [18.43273756128771]
  Quantum machine learning (QML) has emerged as an innovative framework with the potential to uncover complex patterns. We introduce quantum parallel information exchange (QPIE) hybrid network, a new non-sequential hybrid classical quantum model architecture. We develop a dynamic gradient selection method that applies the parameter shift rule on quantum processing units.
  arXiv  Detail & Related papers  (2025-04-05T17:25:26Z)
• Quantum Adaptive Self-Attention for Quantum Transformer Models [0.0]
  We propose Quantum Adaptive Self-Attention (QASA), a novel hybrid architecture that enhances classical Transformer models with a quantum attention mechanism. QASA replaces dot-product attention with a parameterized quantum circuit (PQC) that adaptively captures inter-token relationships in the quantum Hilbert space. Experiments on synthetic time-series tasks demonstrate that QASA achieves faster convergence and superior generalization compared to both standard Transformers and reduced classical variants.
  arXiv  Detail & Related papers  (2025-04-05T02:52:37Z)
• Quantum Convolutional Neural Network with Flexible Stride [7.362858964229726]
  We propose a novel quantum convolutional neural network algorithm. It can flexibly adjust the stride to accommodate different tasks. It can achieve exponential acceleration of data scale in less memory compared with its classical counterpart.
  arXiv  Detail & Related papers  (2024-12-01T02:37:06Z)
• Exploiting recursive structures for the design of novel quantum primitives [0.1227734309612871]
  This paper focuses on generating novel quantum primitives. We show how these structures can be exploited to design new, potentially advantageous quantum algorithms. We comment on the potential impact on quantum algorithms, numerical analysis, and signal processing.
  arXiv  Detail & Related papers  (2024-10-17T17:45:50Z)
• Efficient Learning for Linear Properties of Bounded-Gate Quantum Circuits [63.733312560668274]
  Given a quantum circuit containing d tunable RZ gates and G-d Clifford gates, can a learner perform purely classical inference to efficiently predict its linear properties? We prove that the sample complexity scaling linearly in d is necessary and sufficient to achieve a small prediction error, while the corresponding computational complexity may scale exponentially in d. We devise a kernel-based learning model capable of trading off prediction error and computational complexity, transitioning from exponential to scaling in many practical settings.
  arXiv  Detail & Related papers  (2024-08-22T08:21:28Z)
• PRformer: Pyramidal Recurrent Transformer for Multivariate Time Series Forecasting [82.03373838627606]
  Self-attention mechanism in Transformer architecture requires positional embeddings to encode temporal order in time series prediction. We argue that this reliance on positional embeddings restricts the Transformer's ability to effectively represent temporal sequences. We present a model integrating PRE with a standard Transformer encoder, demonstrating state-of-the-art performance on various real-world datasets.
  arXiv  Detail & Related papers  (2024-08-20T01:56:07Z)
• Zero and Finite Temperature Quantum Simulations Powered by Quantum Magic [1.5998200006932823]
  We introduce a quantum information theory-inspired method to improve the characterization of many-body Hamiltonians on near-term quantum devices. We show that our protocol leads to significant performance improvements for zero and finite temperature free energy calculations on both digital and analog quantum hardware.
  arXiv  Detail & Related papers  (2023-08-22T17:59:41Z)
• TeD-Q: a tensor network enhanced distributed hybrid quantum machine learning framework [59.07246314484875]
  TeD-Q is an open-source software framework for quantum machine learning. It seamlessly integrates classical machine learning libraries with quantum simulators. It provides a graphical mode in which the quantum circuit and the training progress can be visualized in real-time.
  arXiv  Detail & Related papers  (2023-01-13T09:35:05Z)
• The Quantum Path Kernel: a Generalized Quantum Neural Tangent Kernel for Deep Quantum Machine Learning [52.77024349608834]
  Building a quantum analog of classical deep neural networks represents a fundamental challenge in quantum computing. Key issue is how to address the inherent non-linearity of classical deep learning. We introduce the Quantum Path Kernel, a formulation of quantum machine learning capable of replicating those aspects of deep machine learning.
  arXiv  Detail & Related papers  (2022-12-22T16:06:24Z)
• Fock-space Schrieffer--Wolff transformation: classically-assisted rank-reduced quantum phase estimation algorithm [0.0]
  In this paper, we focus on the Schrieffer--Wolff (SW) transformation of the electronic Hamiltonians for molecular systems. We demonstrate that by employing Fock-space variants of the SW transformation one can significantly increase the locality of the qubit-mapped similarity transformed Hamiltonians. The RRST formalism serves as a design principle for developing new classes of approximate schemes that reduce the complexity of quantum circuits.
  arXiv  Detail & Related papers  (2022-11-18T23:06:57Z)
• Transformer variational wave functions for frustrated quantum spin systems [0.0]
  We propose an adaptation of the ViT architecture with complex parameters to define a new class of variational neural-network states. The success of the ViT wave function relies on mixing both local and global operations.
  arXiv  Detail & Related papers  (2022-11-10T11:56:44Z)
• Quantum Phase Processing and its Applications in Estimating Phase and Entropies [10.8525801756287]
  "quantum phase processing" can directly apply arbitrary trigonometric transformations to eigenphases of a unitary operator. Quantum phase processing can extract the eigen-information of quantum systems by simply measuring the ancilla qubit. We propose a new quantum phase estimation algorithm without quantum Fourier transform, which requires the fewest ancilla qubits and matches the best performance so far.
  arXiv  Detail & Related papers  (2022-09-28T17:41:19Z)
• Vision Transformer with Convolutions Architecture Search [72.70461709267497]
  We propose an architecture search method-Vision Transformer with Convolutions Architecture Search (VTCAS) The high-performance backbone network searched by VTCAS introduces the desirable features of convolutional neural networks into the Transformer architecture. It enhances the robustness of the neural network for object recognition, especially in the low illumination indoor scene.
  arXiv  Detail & Related papers  (2022-03-20T02:59:51Z)
• CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
  This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning. The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery. The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
  arXiv  Detail & Related papers  (2021-12-31T04:37:11Z)
• Quantum algorithms for quantum dynamics: A performance study on the spin-boson model [68.8204255655161]
  Quantum algorithms for quantum dynamics simulations are traditionally based on implementing a Trotter-approximation of the time-evolution operator. variational quantum algorithms have become an indispensable alternative, enabling small-scale simulations on present-day hardware. We show that, despite providing a clear reduction of quantum gate cost, the variational method in its current implementation is unlikely to lead to a quantum advantage.
  arXiv  Detail & Related papers  (2021-08-09T18:00:05Z)
• Visual Saliency Transformer [127.33678448761599]
  We develop a novel unified model based on a pure transformer, Visual Saliency Transformer (VST), for both RGB and RGB-D salient object detection (SOD) It takes image patches as inputs and leverages the transformer to propagate global contexts among image patches. Experimental results show that our model outperforms existing state-of-the-art results on both RGB and RGB-D SOD benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-25T08:24:06Z)
• Transformers Solve the Limited Receptive Field for Monocular Depth Prediction [82.90445525977904]
  We propose TransDepth, an architecture which benefits from both convolutional neural networks and transformers. This is the first paper which applies transformers into pixel-wise prediction problems involving continuous labels.
  arXiv  Detail & Related papers  (2021-03-22T18:00:13Z)
• Information Scrambling in Computationally Complex Quantum Circuits [56.22772134614514]
  We experimentally investigate the dynamics of quantum scrambling on a 53-qubit quantum processor. We show that while operator spreading is captured by an efficient classical model, operator entanglement requires exponentially scaled computational resources to simulate.
  arXiv  Detail & Related papers  (2021-01-21T22:18:49Z)
• Generation of High-Resolution Handwritten Digits with an Ion-Trap Quantum Computer [55.41644538483948]
  We implement a quantum-circuit based generative model to learn and sample the prior distribution of a Generative Adversarial Network. We train this hybrid algorithm on an ion-trap device based on $171$Yb$+$ ion qubits to generate high-quality images.
  arXiv  Detail & Related papers  (2020-12-07T18:51:28Z)
• Experimental Quantum Generative Adversarial Networks for Image Generation [93.06926114985761]
  We experimentally achieve the learning and generation of real-world hand-written digit images on a superconducting quantum processor. Our work provides guidance for developing advanced quantum generative models on near-term quantum devices.
  arXiv  Detail & Related papers  (2020-10-13T06:57:17Z)
• Autoregressive Transformer Neural Network for Simulating Open Quantum Systems via a Probabilistic Formulation [5.668795025564699]
  We present an approach for tackling open quantum system dynamics. We compactly represent quantum states with autoregressive transformer neural networks. Efficient algorithms have been developed to simulate the dynamics of the Liouvillian superoperator.
  arXiv  Detail & Related papers  (2020-09-11T18:00:00Z)
• Recurrent Quantum Neural Networks [7.6146285961466]
  Recurrent neural networks are the foundation of many sequence-to-sequence models in machine learning. We construct a quantum recurrent neural network (QRNN) with demonstrable performance on non-trivial tasks. We evaluate the QRNN on MNIST classification, both by feeding the QRNN each image pixel-by-pixel; and by utilising modern data augmentation as preprocessing step.
  arXiv  Detail & Related papers  (2020-06-25T17:59:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.