Neural Tangent Kernel of Matrix Product States: Convergence and Applications

• URL: http://arxiv.org/abs/2111.14046v1
• Date: Sun, 28 Nov 2021 04:10:06 GMT
• Title: Neural Tangent Kernel of Matrix Product States: Convergence and Applications
• Authors: Erdong Guo, David Draper
• Abstract summary: We study the Neural Tangent Kernel (NTK) of Matrix Product States (MPS) We prove that the NTK of MPSally converges to a constant matrix during the gradient descent (training) process. We analyze their dynamics in the infinite bond dimensional limit.
• Score: 1.7894377200944511
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we study the Neural Tangent Kernel (NTK) of Matrix Product States (MPS) and the convergence of its NTK in the infinite bond dimensional limit. We prove that the NTK of MPS asymptotically converges to a constant matrix during the gradient descent (training) process (and also the initialization phase) as the bond dimensions of MPS go to infinity by the observation that the variation of the tensors in MPS asymptotically goes to zero during training in the infinite limit. By showing the positive-definiteness of the NTK of MPS, the convergence of MPS during the training in the function space (space of functions represented by MPS) is guaranteed without any extra assumptions of the data set. We then consider the settings of (supervised) Regression with Mean Square Error (RMSE) and (unsupervised) Born Machines (BM) and analyze their dynamics in the infinite bond dimensional limit. The ordinary differential equations (ODEs) which describe the dynamics of the responses of MPS in the RMSE and BM are derived and solved in the closed-form. For the Regression, we consider Mercer Kernels (Gaussian Kernels) and find that the evolution of the mean of the responses of MPS follows the largest eigenvalue of the NTK. Due to the orthogonality of the kernel functions in BM, the evolution of different modes (samples) decouples and the "characteristic time" of convergence in training is obtained.

Related papers

• Theory of free fermions dynamics under partial post-selected monitoring [49.1574468325115]
  We derive a partial post-selected Schrdinger"o equation based on a microscopic description of continuous weak measurement. We show that the passage to the monitored universality occurs abruptly at finite partial post-selection. Our approach establishes a way to study MiPTs for arbitrary subsets of quantum trajectories.
  arXiv  Detail & Related papers  (2023-12-21T16:53:42Z)
• A Functional-Space Mean-Field Theory of Partially-Trained Three-Layer Neural Networks [49.870593940818715]
  We study the infinite-width limit of a type of three-layer NN model whose first layer is random and fixed. Our theory accommodates different scaling choices of the model, resulting in two regimes of the MF limit that demonstrate distinctive behaviors.
  arXiv  Detail & Related papers  (2022-10-28T17:26:27Z)
• Non-equilibrium quantum impurity problems via matrix-product states in the temporal domain [0.0]
  We propose an approach to analyze impurity dynamics based on the matrix-product state (MPS) representation of the Feynman-Vernon influence functional (IF) We obtain explicit expressions of the wave function for a family of one-dimensional reservoirs, and analyze the scaling of TE with the evolution time for different reservoir's initial states. The approach can be applied to a number of experimental setups, including highly non-equilibrium transport via quantum dots and real-time formation of impurity-reservoir correlations.
  arXiv  Detail & Related papers  (2022-05-10T16:05:25Z)
• Noise-resilient Edge Modes on a Chain of Superconducting Qubits [103.93329374521808]
  Inherent symmetry of a quantum system may protect its otherwise fragile states. We implement the one-dimensional kicked Ising model which exhibits non-local Majorana edge modes (MEMs) with $mathbbZ$ parity symmetry. MEMs are found to be resilient against certain symmetry-breaking noise owing to a prethermalization mechanism.
  arXiv  Detail & Related papers  (2022-04-24T22:34:15Z)
• Developments in the Tensor Network -- from Statistical Mechanics to Quantum Entanglement [0.0]
  Review provides a unified description of a series of developments in the TN from the statistical mechanics side. We explain how the corner transfer matrix (CTM) can be evolved to such MPS-based approaches as density matrix renormalization group (DMRG) and infinite time-evolved block decimation. We then discuss how the difficulty in TRGs for critical systems can be overcome in the tensor network renormalization and the multi-scale entanglement renormalization ansatz.
  arXiv  Detail & Related papers  (2021-11-24T01:46:36Z)
• Non-Markovian Stochastic Schr\"odinger Equation: Matrix Product State Approach to the Hierarchy of Pure States [65.25197248984445]
  We derive a hierarchy of matrix product states (HOMPS) for non-Markovian dynamics in open finite temperature. The validity and efficiency of HOMPS is demonstrated for the spin-boson model and long chains where each site is coupled to a structured, strongly non-Markovian environment.
  arXiv  Detail & Related papers  (2021-09-14T01:47:30Z)
• Representation Theorem for Matrix Product States [1.7894377200944511]
  We investigate the universal representation capacity of the Matrix Product States (MPS) from the perspective of functions and continuous functions. We show that MPS can accurately realize arbitrary functions by providing a construction method of the corresponding MPS structure for an arbitrarily given gate. We study the relation between MPS and neural networks and show that the MPS with a scale-invariant sigmoidal function is equivalent to a one-hidden-layer neural network.
  arXiv  Detail & Related papers  (2021-03-15T11:06:54Z)
• A Dynamical Central Limit Theorem for Shallow Neural Networks [48.66103132697071]
  We prove that the fluctuations around the mean limit remain bounded in mean square throughout training. If the mean-field dynamics converges to a measure that interpolates the training data, we prove that the deviation eventually vanishes in the CLT scaling.
  arXiv  Detail & Related papers  (2020-08-21T18:00:50Z)
• The Convergence Indicator: Improved and completely characterized parameter bounds for actual convergence of Particle Swarm Optimization [68.8204255655161]
  We introduce a new convergence indicator that can be used to calculate whether the particles will finally converge to a single point or diverge. Using this convergence indicator we provide the actual bounds completely characterizing parameter regions that lead to a converging swarm.
  arXiv  Detail & Related papers  (2020-06-06T19:08:05Z)
• Variational optimization of continuous matrix product states [0.0]
  We show how to optimize continuous matrix product states for systems with inhomogeneous external potentials. We show how both the energy and its backwards derivative can be calculated exactly and at a cost that scales as the cube of the bond dimension. We illustrate this by finding ground states of interacting bosons in external potentials, and by calculating boundary or Casimir energy corrections of continuous many-body systems.
  arXiv  Detail & Related papers  (2020-06-02T17:35:41Z)
• Spectra of the Conjugate Kernel and Neural Tangent Kernel for linear-width neural networks [22.57374777395746]
  We study the eigenvalue of the Conjugate Neural Kernel and Tangent Kernel associated to feedforward neural networks. We show that the eigenvalue distributions of the CK and NTK converge to deterministic limits.
  arXiv  Detail & Related papers  (2020-05-25T01:11:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.