Derivation of Learning Rules for Coupled Principal Component Analysis in a Lagrange-Newton Framework

• URL: http://arxiv.org/abs/2204.13460v1
• Date: Thu, 28 Apr 2022 12:50:11 GMT
• Title: Derivation of Learning Rules for Coupled Principal Component Analysis in a Lagrange-Newton Framework
• Authors: Ralf M\"oller
• Abstract summary: We describe a Lagrange-Newton framework for the derivation of learning rules with desirable convergence properties. A Newton descent is applied to an extended variable vector which also includes Lagrange multipliers introduced with constraints. The framework produces "coupled" PCA learning rules which simultaneously estimate an eigenvector and the corresponding eigenvalue in cross-coupled differential equations.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We describe a Lagrange-Newton framework for the derivation of learning rules with desirable convergence properties and apply it to the case of principal component analysis (PCA). In this framework, a Newton descent is applied to an extended variable vector which also includes Lagrange multipliers introduced with constraints. The Newton descent guarantees equal convergence speed from all directions, but is also required to produce stable fixed points in the system with the extended state vector. The framework produces "coupled" PCA learning rules which simultaneously estimate an eigenvector and the corresponding eigenvalue in cross-coupled differential equations. We demonstrate the feasibility of this approach for two PCA learning rules, one for the estimation of the principal, the other for the estimate of an arbitrary eigenvector-eigenvalue pair (eigenpair).

Related papers

• From explained variance of correlated components to PCA without orthogonality constraints [0.0]
  Block Principal Component Analysis (Block PCA) of a data matrix A is difficult to use for the design of sparse PCA by 1 regularization. We introduce new objective matrix functions expvar(Y) which measure the part of the variance of the data matrix A explained by correlated components Y = AZ.
  arXiv  Detail & Related papers  (2024-02-07T09:32:32Z)
• General Solution and Canonical Quantization of the Conic Path Constrained Second-Class System [0.0]
  We consider the problem of constrained motion along a conic path under a given external potential function. We perform the canonical quantization in a consistent way in terms of the corresponding Dirac brackets. The complete Dirac brackets algebra in phase space as well as its physical realization in terms of differential operators is explicitly obtained.
  arXiv  Detail & Related papers  (2022-02-15T13:46:56Z)
• Controlling the Complexity and Lipschitz Constant improves polynomial nets [55.121200972539114]
  We derive new complexity bounds for the set of Coupled CP-Decomposition (CCP) and Nested Coupled CP-decomposition (NCP) models of Polynomial Nets. We propose a principled regularization scheme that we evaluate experimentally in six datasets and show that it improves the accuracy as well as the robustness of the models to adversarial perturbations.
  arXiv  Detail & Related papers  (2022-02-10T14:54:29Z)
• Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems [91.3755431537592]
  We consider a control system of the form $dot x = sum_i=1lF_i(x)u_i$, with linear dependence in the controls. We use the corresponding flow to approximate the action of a diffeomorphism on a compact ensemble of points.
  arXiv  Detail & Related papers  (2021-10-24T08:57:46Z)
• Eigen Analysis of Self-Attention and its Reconstruction from Partial Computation [58.80806716024701]
  We study the global structure of attention scores computed using dot-product based self-attention. We find that most of the variation among attention scores lie in a low-dimensional eigenspace. We propose to compute scores only for a partial subset of token pairs, and use them to estimate scores for the remaining pairs.
  arXiv  Detail & Related papers  (2021-06-16T14:38:42Z)
• Signatures of Chaos in Non-integrable Models of Quantum Field Theory [0.0]
  We study signatures of quantum chaos in (1+1)D Quantum Field Theory (QFT) models. We focus on the double sine-Gordon, also studying the massive sine-Gordon and $phi4$ model.
  arXiv  Detail & Related papers  (2020-12-15T18:56:20Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)
• Exponentially Weighted l_2 Regularization Strategy in Constructing Reinforced Second-order Fuzzy Rule-based Model [72.57056258027336]
  In the conventional Takagi-Sugeno-Kang (TSK)-type fuzzy models, constant or linear functions are usually utilized as the consequent parts of the fuzzy rules. We introduce an exponential weight approach inspired by the weight function theory encountered in harmonic analysis.
  arXiv  Detail & Related papers  (2020-07-02T15:42:15Z)
• Derivation of Symmetric PCA Learning Rules from a Novel Objective Function [0.0]
  Neural learning rules for principal component / subspace analysis can be derived by maximizing an objective function. For a subspace with a single axis, the optimization produces the principal eigenvector of the data covariance matrix. For a subspace with multiple axes, the optimization leads to PSA learning rules which only converge to axes spanning the principal subspace but not to the principal eigenvectors.
  arXiv  Detail & Related papers  (2020-05-24T08:57:54Z)
• Derivation of Coupled PCA and SVD Learning Rules from a Newton Zero-Finding Framework [0.0]
  A method to derive coupled learning rules from information criteria by Newton optimization is known. Here we describe an alternative approach where coupled PCA and SVD learning rules can systematically be derived from a Newton zero-finding framework. To demonstrate the framework, we derive PCA and SVD learning rules with constant Euclidean length or constant sum of the vector estimates.
  arXiv  Detail & Related papers  (2020-03-25T15:49:55Z)
• Semiparametric Nonlinear Bipartite Graph Representation Learning with Provable Guarantees [106.91654068632882]
  We consider the bipartite graph and formalize its representation learning problem as a statistical estimation problem of parameters in a semiparametric exponential family distribution. We show that the proposed objective is strongly convex in a neighborhood around the ground truth, so that a gradient descent-based method achieves linear convergence rate. Our estimator is robust to any model misspecification within the exponential family, which is validated in extensive experiments.
  arXiv  Detail & Related papers  (2020-03-02T16:40:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.