Full Encoder: Make Autoencoders Learn Like PCA

• URL: http://arxiv.org/abs/2103.14082v1
• Date: Thu, 25 Mar 2021 18:57:30 GMT
• Title: Full Encoder: Make Autoencoders Learn Like PCA
• Abstract summary: Full is a novel unified autoencoder framework as a correspondence to PCA in the non-linear domain. The latent variables acquired with Full are stable and robust, as they always learn the same representation regardless the network initial states. Full can be used to determine the degrees of freedom in a non-linear system, and is useful for data compression or anomaly detection.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While the beta-VAE family is aiming to find disentangled representations and acquire human-interpretable generative factors, like what an ICA does in the linear domain, we propose Full Encoder: a novel unified autoencoder framework as a correspondence to PCA in the non-linear domain. The idea is to train an autoencoder with one latent variable first, then involve more latent variables progressively to refine the reconstruction results. The latent variables acquired with Full Encoder is stable and robust, as they always learn the same representation regardless the network initial states. Full Encoder can be used to determine the degrees of freedom in a non-linear system, and is useful for data compression or anomaly detection. Full Encoder can also be combined with beta-VAE framework to sort out the importance of the generative factors, providing more insights for non-linear system analysis. We created a toy dataset with a non-linear system to test the Full Encoder and compare its results to VAE and beta-VAE's results.

Related papers

• Partially Linear Autoencoders for Manifold Learning and Dimensionality Reduction [0.0]
  Autoencoders are widely used for nonlinear dimensionality reduction and manifold learning.<n>We investigate the specific role of the encoder and the extent to which it can be constrained to be linear without reducing accuracy.
  arXiv  Detail & Related papers  (2026-08-30T15:54:48Z)
• Transformer-based Koopman Autoencoder for Linearizing Fisher's Equation [0.0]
  A Transformer-based Koopman autoencoder is proposed for linear Fisher's reaction-diffusion equation. The emphasis is on not just solving the equation but also transforming the system's dynamics into a more comprehensible form. Extensive testing on multiple datasets was used to assess the efficacy of the proposed model.
  arXiv  Detail & Related papers  (2024-12-03T12:52:04Z)
• Symmetric Equilibrium Learning of VAEs [56.56929742714685]
  We view variational autoencoders (VAEs) as decoder-encoder pairs, which map distributions in the data space to distributions in the latent space and vice versa. We propose a Nash equilibrium learning approach, which is symmetric with respect to the encoder and decoder and allows learning VAEs in situations where both the data and the latent distributions are accessible only by sampling.
  arXiv  Detail & Related papers  (2023-07-19T10:27:34Z)
• Benign Autoencoders [0.0]
  We formalize the problem of finding the optimal encoder-decoder pair and characterize its solution, which we name the "benign autoencoder" (BAE) We prove that BAE projects data onto a manifold whose dimension is the optimal compressibility dimension of the generative problem. As an illustration, we show how BAE can find optimal, low-dimensional latent representations that improve the performance of a discriminator under a distribution shift.
  arXiv  Detail & Related papers  (2022-10-02T21:36:27Z)
• String-based Molecule Generation via Multi-decoder VAE [56.465033997245776]
  We investigate the problem of string-based molecular generation via variational autoencoders (VAEs) We propose a simple, yet effective idea to improve the performance of VAE for the task. In our experiments, the proposed VAE model particularly performs well for generating a sample from out-of-domain distribution.
  arXiv  Detail & Related papers  (2022-08-23T03:56:30Z)
• Embrace the Gap: VAEs Perform Independent Mechanism Analysis [36.686468842036305]
  We study nonlinear VAEs in the limit of near-deterministic decoders. We show that VAEs uncover the true latent factors when the data generating process satisfies the IMA assumption.
  arXiv  Detail & Related papers  (2022-06-06T08:19:19Z)
• Closing the gap: Exact maximum likelihood training of generative autoencoders using invertible layers [7.76925617801895]
  We show that VAE-style autoencoders can be constructed using invertible layers, which offer a tractable exact likelihood without the need for regularization terms. This is achieved while leaving complete freedom in the choice of encoder, decoder and prior architectures. We show that the approach results in strikingly higher performance than architecturally equivalent VAEs in term of log-likelihood, sample quality and denoising performance.
  arXiv  Detail & Related papers  (2022-05-19T13:16:09Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)
• End-to-End Synthetic Data Generation for Domain Adaptation of Question Answering Systems [34.927828428293864]
  Our model comprises a single transformer-based encoder-decoder network that is trained end-to-end to generate both answers and questions. In a nutshell, we feed a passage to the encoder and ask the decoder to generate a question and an answer token-by-token.
  arXiv  Detail & Related papers  (2020-10-12T21:10:18Z)
• Simple and Effective VAE Training with Calibrated Decoders [123.08908889310258]
  Variational autoencoders (VAEs) provide an effective and simple method for modeling complex distributions. We study the impact of calibrated decoders, which learn the uncertainty of the decoding distribution. We propose a simple but novel modification to the commonly used Gaussian decoder, which computes the prediction variance analytically.
  arXiv  Detail & Related papers  (2020-06-23T17:57:47Z)
• On the Encoder-Decoder Incompatibility in Variational Text Modeling and Beyond [82.18770740564642]
  Variational autoencoders (VAEs) combine latent variables with amortized variational inference. We observe the encoder-decoder incompatibility that leads to poor parameterizations of the data manifold. We propose Coupled-VAE, which couples a VAE model with a deterministic autoencoder with the same structure.
  arXiv  Detail & Related papers  (2020-04-20T10:34:10Z)
• Learning Autoencoders with Relational Regularization [89.53065887608088]
  A new framework is proposed for learning autoencoders of data distributions. We minimize the discrepancy between the model and target distributions, with a emphrelational regularization We implement the framework with two scalable algorithms, making it applicable for both probabilistic and deterministic autoencoders.
  arXiv  Detail & Related papers  (2020-02-07T17:27:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.