Learning Ordered Representations in Latent Space for Intrinsic Dimension Estimation via Principal Component Autoencoder

• URL: http://arxiv.org/abs/2601.19179v1
• Date: Tue, 27 Jan 2026 04:24:21 GMT
• Title: Learning Ordered Representations in Latent Space for Intrinsic Dimension Estimation via Principal Component Autoencoder
• Authors: Qipeng Zhan, Zhuoping Zhou, Zexuan Wang, Li Shen,
• Abstract summary: Autoencoders have long been considered a nonlinear extension of Principal Component Analysis (PCA)<n>We propose a novel autoencoder framework that integrates non-uniform variance regularization with an isometric constraint.<n>This design serves as a natural generalization of PCA, enabling the model to preserve key advantages, such as ordered representations and variance retention.
• Score: 10.509144950561103
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoencoders have long been considered a nonlinear extension of Principal Component Analysis (PCA). Prior studies have demonstrated that linear autoencoders (LAEs) can recover the ordered, axis-aligned principal components of PCA by incorporating non-uniform $\ell_2$ regularization or by adjusting the loss function. However, these approaches become insufficient in the nonlinear setting, as the remaining variance cannot be properly captured independently of the nonlinear mapping. In this work, we propose a novel autoencoder framework that integrates non-uniform variance regularization with an isometric constraint. This design serves as a natural generalization of PCA, enabling the model to preserve key advantages, such as ordered representations and variance retention, while remaining effective for nonlinear dimensionality reduction tasks.

Related papers

• MirrorLA: Reflecting Feature Map for Vision Linear Attention [49.41670925034762]
  Linear attention significantly reduces the computational complexity of Transformers from quadratic to linear, yet it consistently lags behind softmax-based attention in performance.<n>We propose MirrorLA, a geometric framework that substitutes passive truncation with active reorientation.<n>MirrorLA achieves state-of-the-art performance across standard benchmarks, demonstrating that strictly linear efficiency can be achieved without compromising representational fidelity.
  arXiv  Detail & Related papers  (2026-02-04T09:14:09Z)
• Beyond Additivity: Sparse Isotonic Shapley Regression toward Nonlinear Explainability [0.0]
  We introduce Sparse Isotonic Shapley Regression (SISR), a unified nonlinear explanation framework.<n>SISR learns a monotonic transformation to restore additivity--obviating the need for a closed-form specification--and enforces an L0 sparsity constraint on the Shapley vector.<n>SISR stabilizes attributions across payoff schemes, correctly filters irrelevant features while standard Shapley values suffer severe rank and sign distortions.
  arXiv  Detail & Related papers  (2025-12-02T08:34:43Z)
• A Random Matrix Analysis of In-context Memorization for Nonlinear Attention [18.90197287760915]
  We show that nonlinear Attention incurs higher memorization error than linear ridge regression on random inputs.<n>Our results reveal how nonlinearity and input structure interact with each other to govern the memorization performance of nonlinear Attention.
  arXiv  Detail & Related papers  (2025-06-23T13:56:43Z)
• Adversarial Dependence Minimization [78.36795688238155]
  This work provides a differentiable and scalable algorithm for dependence minimization that goes beyond linear pairwise decorrelation.<n>We demonstrate its utility in three applications: extending PCA to nonlinear decorrelation, improving the generalization of image classification methods, and preventing dimensional collapse in self-supervised representation learning.
  arXiv  Detail & Related papers  (2025-02-05T14:43:40Z)
• Refined Risk Bounds for Unbounded Losses via Transductive Priors [67.12679195076387]
  We revisit the sequential variants of linear regression with the squared loss, classification problems with hinge loss, and logistic regression.<n>Our key tools are based on the exponential weights algorithm with carefully chosen transductive priors.
  arXiv  Detail & Related papers  (2024-10-29T00:01:04Z)
• Controlled Learning of Pointwise Nonlinearities in Neural-Network-Like Architectures [14.93489065234423]
  We present a general variational framework for the training of freeform nonlinearities in layered computational architectures.<n>The slope constraints allow us to impose properties such as 1-Lipschitz stability, firm non-expansiveness, and monotonicity/invertibility.<n>We show how to solve the numerically function-optimization problem by representing the nonlinearities in a suitable (nonuniform) B-spline basis.
  arXiv  Detail & Related papers  (2024-08-23T14:39:27Z)
• $σ$-PCA: a building block for neural learning of identifiable linear transformations [0.0]
  $sigma$-PCA is a method that formulates a unified model for linear and nonlinear PCA. nonlinear PCA can be seen as a method that maximizes both variance and statistical independence.
  arXiv  Detail & Related papers  (2023-11-22T18:34:49Z)
• Gram-Schmidt Methods for Unsupervised Feature Extraction and Selection [7.373617024876725]
  We propose a Gram-Schmidt process over function spaces to detect and map out nonlinear dependencies.<n>We provide experimental results for synthetic and real-world benchmark datasets.<n>Surprisingly, our linear feature extraction algorithms are comparable and often outperform several important nonlinear feature extraction methods.
  arXiv  Detail & Related papers  (2023-11-15T21:29:57Z)
• Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods [91.54785981649228]
  This paper focuses on non-linear two-layer autoencoders trained in the challenging proportional regime. Our results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods. For the special case of a sign activation function, our analysis establishes the fundamental limits for the lossy compression of Gaussian sources via (shallow) autoencoders.
  arXiv  Detail & Related papers  (2022-12-27T12:37:34Z)
• PCA-Boosted Autoencoders for Nonlinear Dimensionality Reduction in Low Data Regimes [0.2925461470287228]
  We propose a technique that harnesses the best of both worlds: an autoencoder that leverages PCA to perform well on scarce nonlinear data. A synthetic example is presented first to study the effects of data nonlinearity and size on the performance of the proposed method.
  arXiv  Detail & Related papers  (2022-05-23T23:46:52Z)
• LQF: Linear Quadratic Fine-Tuning [114.3840147070712]
  We present the first method for linearizing a pre-trained model that achieves comparable performance to non-linear fine-tuning. LQF consists of simple modifications to the architecture, loss function and optimization typically used for classification.
  arXiv  Detail & Related papers  (2020-12-21T06:40:20Z)
• Sparse Quantized Spectral Clustering [85.77233010209368]
  We exploit tools from random matrix theory to make precise statements about how the eigenspectrum of a matrix changes under such nonlinear transformations. We show that very little change occurs in the informative eigenstructure even under drastic sparsification/quantization.
  arXiv  Detail & Related papers  (2020-10-03T15:58:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.