$σ$-PCA: a building block for neural learning of identifiable linear transformations

• URL: http://arxiv.org/abs/2311.13580v4
• Date: Mon, 1 Jul 2024 09:55:40 GMT
• Title: $σ$-PCA: a building block for neural learning of identifiable linear transformations
• Authors: Fahdi Kanavati, Lucy Katsnith, Masayuki Tsuneki,
• Abstract summary: $sigma$-PCA is a method that formulates a unified model for linear and nonlinear PCA. nonlinear PCA can be seen as a method that maximizes both variance and statistical independence.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Linear principal component analysis (PCA) learns (semi-)orthogonal transformations by orienting the axes to maximize variance. Consequently, it can only identify orthogonal axes whose variances are clearly distinct, but it cannot identify the subsets of axes whose variances are roughly equal. It cannot eliminate the subspace rotational indeterminacy: it fails to disentangle components with equal variances (eigenvalues), resulting, in each eigen subspace, in randomly rotated axes. In this paper, we propose $\sigma$-PCA, a method that (1) formulates a unified model for linear and nonlinear PCA, the latter being a special case of linear independent component analysis (ICA), and (2) introduces a missing piece into nonlinear PCA that allows it to eliminate, from the canonical linear PCA solution, the subspace rotational indeterminacy -- without whitening the inputs. Whitening, a preprocessing step which converts the inputs into unit-variance inputs, has generally been a prerequisite step for linear ICA methods, which meant that conventional nonlinear PCA could not necessarily preserve the orthogonality of the overall transformation, could not directly reduce dimensionality, and could not intrinsically order by variances. We offer insights on the relationship between linear PCA, nonlinear PCA, and linear ICA -- three methods with autoencoder formulations for learning special linear transformations from data, transformations that are (semi-)orthogonal for PCA, and arbitrary unit-variance for ICA. As part of our formulation, nonlinear PCA can be seen as a method that maximizes both variance and statistical independence, lying in the middle between linear PCA and linear ICA, serving as a building block for learning linear transformations that are identifiable.

Related papers

• RLE: A Unified Perspective of Data Augmentation for Cross-Spectral Re-identification [59.5042031913258]
  Non-linear modality discrepancy mainly comes from diverse linear transformations acting on the surface of different materials. We propose a Random Linear Enhancement (RLE) strategy which includes Moderate Random Linear Enhancement (MRLE) and Radical Random Linear Enhancement (RRLE) The experimental results not only demonstrate the superiority and effectiveness of RLE but also confirm its great potential as a general-purpose data augmentation for cross-spectral re-identification.
  arXiv  Detail & Related papers  (2024-11-02T12:13:37Z)
• From explained variance of correlated components to PCA without orthogonality constraints [0.0]
  Block Principal Component Analysis (Block PCA) of a data matrix A is difficult to use for the design of sparse PCA by 1 regularization. We introduce new objective matrix functions expvar(Y) which measure the part of the variance of the data matrix A explained by correlated components Y = AZ.
  arXiv  Detail & Related papers  (2024-02-07T09:32:32Z)
• Discrete-Time Nonlinear Feedback Linearization via Physics-Informed Machine Learning [0.0]
  We present a physics-informed machine learning scheme for the feedback linearization of nonlinear systems. We show that the proposed PIML outperforms the traditional numerical implementation.
  arXiv  Detail & Related papers  (2023-03-15T19:03:23Z)
• Linear Convergence of Natural Policy Gradient Methods with Log-Linear Policies [115.86431674214282]
  We consider infinite-horizon discounted Markov decision processes and study the convergence rates of the natural policy gradient (NPG) and the Q-NPG methods with the log-linear policy class. We show that both methods attain linear convergence rates and $mathcalO (1/epsilon2)$ sample complexities using a simple, non-adaptive geometrically increasing step size.
  arXiv  Detail & Related papers  (2022-10-04T06:17:52Z)
• PCA-Boosted Autoencoders for Nonlinear Dimensionality Reduction in Low Data Regimes [0.2925461470287228]
  We propose a technique that harnesses the best of both worlds: an autoencoder that leverages PCA to perform well on scarce nonlinear data. A synthetic example is presented first to study the effects of data nonlinearity and size on the performance of the proposed method.
  arXiv  Detail & Related papers  (2022-05-23T23:46:52Z)
• Theoretical Connection between Locally Linear Embedding, Factor Analysis, and Probabilistic PCA [13.753161236029328]
  Linear Embedding (LLE) is a nonlinear spectral dimensionality reduction and manifold learning method. In this work, we look at the linear reconstruction step from a perspective where it is assumed that every data point is conditioned on its linear reconstruction weights as latent factors.
  arXiv  Detail & Related papers  (2022-03-25T21:07:20Z)
• Nonlinear Isometric Manifold Learning for Injective Normalizing Flows [58.720142291102135]
  We use isometries to separate manifold learning and density estimation. We also employ autoencoders to design embeddings with explicit inverses that do not distort the probability distribution.
  arXiv  Detail & Related papers  (2022-03-08T08:57:43Z)
• Symmetry-Aware Autoencoders: s-PCA and s-nlPCA [0.0]
  We introduce a novel machine learning embedding in the autoencoder, which uses spatial transformer networks and Siamese networks to account for continuous and discrete symmetries. The proposed symmetry-aware autoencoder is invariant to predetermined input transformations dictating the dynamics of the underlying physical system.
  arXiv  Detail & Related papers  (2021-11-04T14:22:19Z)
• Unfolding Projection-free SDP Relaxation of Binary Graph Classifier via GDPA Linearization [59.87663954467815]
  Algorithm unfolding creates an interpretable and parsimonious neural network architecture by implementing each iteration of a model-based algorithm as a neural layer. In this paper, leveraging a recent linear algebraic theorem called Gershgorin disc perfect alignment (GDPA), we unroll a projection-free algorithm for semi-definite programming relaxation (SDR) of a binary graph. Experimental results show that our unrolled network outperformed pure model-based graph classifiers, and achieved comparable performance to pure data-driven networks but using far fewer parameters.
  arXiv  Detail & Related papers  (2021-09-10T07:01:15Z)
• Quantitative Understanding of VAE as a Non-linearly Scaled Isometric Embedding [52.48298164494608]
  Variational autoencoder (VAE) estimates the posterior parameters of latent variables corresponding to each input data. This paper provides a quantitative understanding of VAE property through the differential geometric and information-theoretic interpretations of VAE.
  arXiv  Detail & Related papers  (2020-07-30T02:37:46Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.