Partially Linear Autoencoders for Manifold Learning and Dimensionality Reduction

• URL: http://arxiv.org/abs/2608.29867v1
• Date: Sun, 30 Aug 2026 15:54:48 GMT
• Title: Partially Linear Autoencoders for Manifold Learning and Dimensionality Reduction
• Abstract summary: Autoencoders are widely used for nonlinear dimensionality reduction and manifold learning.<n>We investigate the specific role of the encoder and the extent to which it can be constrained to be linear without reducing accuracy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Autoencoders are widely used for nonlinear dimensionality reduction and manifold learning. While most common implementations rely on both nonlinear encoders and decoders, we investigate the specific role of the encoder and the extent to which it can be constrained to be linear without reducing accuracy. We conduct a comparative study on four autoencoder architectures: standard fully nonlinear autoencoders (AE), linear-encoder autoencoders (Lenc-AE), linear-decoder autoencoders (Ldec-AE), and fully linear autoencoders (LAE), evaluated on synthetic manifolds, computational mechanics data sets, and real-world image data sets including MNIST. We demonstrate that imposing a linear encoder preserves most of the representational capacity of the autoencoder, provided the decoder remains nonlinear. In particular, Lenc-AE consistently outperforms both Ldec-AE and LAE, and achieves reconstruction quality comparable to fully nonlinear AE, while offering advantages in terms of parsimony and interpretability of the latent representation. These results suggest that the nonlinear decoder is the critical component for manifold learning, rather than the encoder. A geometric interpretation of this finding is developed, which identifies the precise conditions under which a linear encoder is sufficient, and the specific manifold configurations that expose its limitations.

Related papers

• Beyond Encoder Accumulation: Measuring Encoder Roles in Multi-Encoder VLMs [50.3757247538081]
  We retrain and evaluate all 31 non-empty subsets of five common vision encoders under a unified pipeline.<n>We decompose each encoder's contribution into two axes, Capacity and Necessity.<n>Per-encoder pre-projector effective rank explains the residual score variation.
  arXiv  Detail & Related papers  (2026-06-02T16:46:42Z)
• Investigating Redundancy in Multimodal Large Language Models with Multiple Vision Encoders [54.03921875874111]
  We show that performance degrades gracefully when selected encoders are masked, revealing pervasive encoder redundancy.<n>We observe strong specialization on tasks like OCR and Chart, where a single encoder can dominate with a CUR greater than 90%.<n>Masking specific encoders can yield up to 16% higher accuracy on a specific task category and 3.6% overall performance boost compared to the full model.
  arXiv  Detail & Related papers  (2025-07-04T02:38:59Z)
• Leveraging time and parameters for nonlinear model reduction methods [0.0]
  We consider problems with slowly decaying Kolmogorov $n$-widths as wave-like or transport-dominated problems.<n>To overcome this Kolmogorov barrier within MOR, nonlinear projections are used, which are often realized numerically using autoencoders.<n>We show that extending the to-be-reduced system and its corresponding training data makes it possible to replace the nonlinear encoder with a linear encoder without sacrificing accuracy.
  arXiv  Detail & Related papers  (2025-01-07T15:10:07Z)
• Rank Reduction Autoencoders [3.180674374101366]
  We introduce a new class of deterministic autoencoders, Rank Reduction Autoencoders (RRAEs)<n>In RRAEs, the bottleneck is defined by the rank of the latent matrix, thereby alleviating the dependence of the encoder/decoder architecture on the bottleneck size.<n>We empirically demonstrate that both RRAEs and aRRAEs are stable, scalable, and reliable.
  arXiv  Detail & Related papers  (2024-05-22T20:33:09Z)
• Learning Linear Block Error Correction Codes [62.25533750469467]
  We propose for the first time a unified encoder-decoder training of binary linear block codes. We also propose a novel Transformer model in which the self-attention masking is performed in a differentiable fashion for the efficient backpropagation of the code gradient.
  arXiv  Detail & Related papers  (2024-05-07T06:47:12Z)
• It's Enough: Relaxing Diagonal Constraints in Linear Autoencoders for Recommendation [4.8802420827610025]
  This paper aims to theoretically understand the properties of two terms in linear autoencoders. We propose simple-yet-effective linear autoencoder models using diagonal inequality constraints, called Relaxed Linear AutoEncoder (RLAE) and Relaxed Denoising Linear AutoEncoder (RDLAE) Experimental results demonstrate that our models are comparable or superior to state-of-the-art linear and non-linear models on six benchmark datasets.
  arXiv  Detail & Related papers  (2023-05-22T11:09:49Z)
• Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods [91.54785981649228]
  This paper focuses on non-linear two-layer autoencoders trained in the challenging proportional regime. Our results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods. For the special case of a sign activation function, our analysis establishes the fundamental limits for the lossy compression of Gaussian sources via (shallow) autoencoders.
  arXiv  Detail & Related papers  (2022-12-27T12:37:34Z)
• Semi-Supervised Manifold Learning with Complexity Decoupled Chart Autoencoders [45.29194877564103]
  This work introduces a chart autoencoder with an asymmetric encoding-decoding process that can incorporate additional semi-supervised information such as class labels. We discuss the approximation power of such networks and derive a bound that essentially depends on the intrinsic dimension of the data manifold rather than the dimension of ambient space.
  arXiv  Detail & Related papers  (2022-08-22T19:58:03Z)
• Variational Autoencoders for Studying the Manifold of Precoding Matrices with High Spectral Efficiency [47.187609203210705]
  We look at how to use a variational autoencoder to find a precoding matrix with a high Spectral Efficiency (SE) Our objective is to create a less time-consuming algorithm with minimum quality degradation.
  arXiv  Detail & Related papers  (2021-11-23T11:45:45Z)
• On the Regularization of Autoencoders [14.46779433267854]
  We show that the unsupervised setting by itself induces strong additional regularization, i.e., a severe reduction in the model-capacity of the learned autoencoder. We derive that a deep nonlinear autoencoder cannot fit the training data more accurately than a linear autoencoder does if both models have the same dimensionality in their last layer. We demonstrate that it is an accurate approximation across all model-ranks in our experiments on three well-known data sets.
  arXiv  Detail & Related papers  (2021-10-21T18:28:25Z)
• Neural Distributed Source Coding [59.630059301226474]
  We present a framework for lossy DSC that is agnostic to the correlation structure and can scale to high dimensions. We evaluate our method on multiple datasets and show that our method can handle complex correlations and state-of-the-art PSNR.
  arXiv  Detail & Related papers  (2021-06-05T04:50:43Z)
• Full Encoder: Make Autoencoders Learn Like PCA [0.0]
  Full is a novel unified autoencoder framework as a correspondence to PCA in the non-linear domain. The latent variables acquired with Full are stable and robust, as they always learn the same representation regardless the network initial states. Full can be used to determine the degrees of freedom in a non-linear system, and is useful for data compression or anomaly detection.
  arXiv  Detail & Related papers  (2021-03-25T18:57:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.