Eigenvalues of Autoencoders in Training and at Initialization

• URL: http://arxiv.org/abs/2201.11813v1
• Date: Thu, 27 Jan 2022 21:34:49 GMT
• Title: Eigenvalues of Autoencoders in Training and at Initialization
• Authors: Benjamin Dees, Susama Agarwala, Corey Lowman
• Abstract summary: We study the distribution of eigenvalues of Jacobian matrices of autoencoders early in the training process. We find that autoencoders that have not been trained have eigenvalue distributions that are qualitatively different from those which have been trained for a long time.
• Score: 0.2578242050187029
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we investigate the evolution of autoencoders near their initialization. In particular, we study the distribution of the eigenvalues of the Jacobian matrices of autoencoders early in the training process, training on the MNIST data set. We find that autoencoders that have not been trained have eigenvalue distributions that are qualitatively different from those which have been trained for a long time ($>$100 epochs). Additionally, we find that even at early epochs, these eigenvalue distributions rapidly become qualitatively similar to those of the fully trained autoencoders. We also compare the eigenvalues at initialization to pertinent theoretical work on the eigenvalues of random matrices and the products of such matrices.

Related papers

• PEOPL: Characterizing Privately Encoded Open Datasets with Public Labels [59.66777287810985]
  We introduce information-theoretic scores for privacy and utility, which quantify the average performance of an unfaithful user. We then theoretically characterize primitives in building families of encoding schemes that motivate the use of random deep neural networks.
  arXiv  Detail & Related papers  (2023-03-31T18:03:53Z)
• Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods [91.54785981649228]
  This paper focuses on non-linear two-layer autoencoders trained in the challenging proportional regime. Our results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods. For the special case of a sign activation function, our analysis establishes the fundamental limits for the lossy compression of Gaussian sources via (shallow) autoencoders.
  arXiv  Detail & Related papers  (2022-12-27T12:37:34Z)
• Eigenvalue initialisation and regularisation for Koopman autoencoders [2.260916274164351]
  We study the Koopman autoencoder model which includes an encoder, a Koopman operator layer, and a decoder. We propose the "eigeninit" initialisation scheme that samples initial Koopman operators from specific eigenvalue distributions. In addition, we suggest the "eigenloss" penalty scheme that penalises the eigenvalues of the Koopman operator during training.
  arXiv  Detail & Related papers  (2022-12-23T00:05:26Z)
• Variational Autoencoders for Studying the Manifold of Precoding Matrices with High Spectral Efficiency [47.187609203210705]
  We look at how to use a variational autoencoder to find a precoding matrix with a high Spectral Efficiency (SE) Our objective is to create a less time-consuming algorithm with minimum quality degradation.
  arXiv  Detail & Related papers  (2021-11-23T11:45:45Z)
• Geometry and Generalization: Eigenvalues as predictors of where a network will fail to generalize [0.30586855806896046]
  We study the deformation of the input space by a trained autoencoder via the Jacobians of the trained weight matrices. This is a dataset independent means of testing an autoencoder's ability to generalize on new input.
  arXiv  Detail & Related papers  (2021-07-13T21:03:42Z)
• Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks [7.412225511828064]
  Deep linear networks trained with gradient descent yield low rank solutions. We show greedy learning of low-rank latent codes induced by a linear sub-network at the autoencoder bottleneck.
  arXiv  Detail & Related papers  (2021-07-02T23:17:50Z)
• Autoencoding Variational Autoencoder [56.05008520271406]
  We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency. We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
  arXiv  Detail & Related papers  (2020-12-07T14:16:14Z)
• On Coresets for Support Vector Machines [61.928187390362176]
  A coreset is a small, representative subset of the original data points. We show that our algorithm can be used to extend the applicability of any off-the-shelf SVM solver to streaming, distributed, and dynamic data settings.
  arXiv  Detail & Related papers  (2020-02-15T23:25:12Z)
• Learning Autoencoders with Relational Regularization [89.53065887608088]
  A new framework is proposed for learning autoencoders of data distributions. We minimize the discrepancy between the model and target distributions, with a emphrelational regularization We implement the framework with two scalable algorithms, making it applicable for both probabilistic and deterministic autoencoders.
  arXiv  Detail & Related papers  (2020-02-07T17:27:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.