Implicit Rank-Minimizing Autoencoder

• URL: http://arxiv.org/abs/2010.00679v2
• Date: Wed, 14 Oct 2020 15:36:27 GMT
• Title: Implicit Rank-Minimizing Autoencoder
• Authors: Li Jing, Jure Zbontar, Yann LeCun
• Abstract summary: Implicit Rank-Minimizing Autoencoder (IRMAE) is simple, deterministic, and learns compact latent spaces. We demonstrate the validity of the method on several image generation and representation learning tasks.
• Score: 21.2045061949013
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: An important component of autoencoders is the method by which the information capacity of the latent representation is minimized or limited. In this work, the rank of the covariance matrix of the codes is implicitly minimized by relying on the fact that gradient descent learning in multi-layer linear networks leads to minimum-rank solutions. By inserting a number of extra linear layers between the encoder and the decoder, the system spontaneously learns representations with a low effective dimension. The model, dubbed Implicit Rank-Minimizing Autoencoder (IRMAE), is simple, deterministic, and learns compact latent spaces. We demonstrate the validity of the method on several image generation and representation learning tasks.

Related papers

• Learning Low-Rank Latent Spaces with Simple Deterministic Autoencoder: Theoretical and Empirical Insights [1.246305060872372]
  Low-Rank Autoencoder (LoRAE) is a simple autoencoder extension that learns low-rank latent space. Our model's superiority shines through various tasks such as image generation and downstream classification.
  arXiv  Detail & Related papers  (2023-10-24T21:24:27Z)
• Divided Attention: Unsupervised Multi-Object Discovery with Contextually Separated Slots [78.23772771485635]
  We introduce a method to segment the visual field into independently moving regions, trained with no ground truth or supervision. It consists of an adversarial conditional encoder-decoder architecture based on Slot Attention.
  arXiv  Detail & Related papers  (2023-04-04T00:26:13Z)
• Fundamental Limits of Two-layer Autoencoders, and Achieving Them with Gradient Methods [91.54785981649228]
  This paper focuses on non-linear two-layer autoencoders trained in the challenging proportional regime. Our results characterize the minimizers of the population risk, and show that such minimizers are achieved by gradient methods. For the special case of a sign activation function, our analysis establishes the fundamental limits for the lossy compression of Gaussian sources via (shallow) autoencoders.
  arXiv  Detail & Related papers  (2022-12-27T12:37:34Z)
• Reducing Redundancy in the Bottleneck Representation of the Autoencoders [98.78384185493624]
  Autoencoders are a type of unsupervised neural networks, which can be used to solve various tasks. We propose a scheme to explicitly penalize feature redundancies in the bottleneck representation. We tested our approach across different tasks: dimensionality reduction using three different dataset, image compression using the MNIST dataset, and image denoising using fashion MNIST.
  arXiv  Detail & Related papers  (2022-02-09T18:48:02Z)
• Closed-Loop Data Transcription to an LDR via Minimaxing Rate Reduction [27.020835928724775]
  This work proposes a new computational framework for learning an explicit generative model for real-world datasets. In particular, we propose to learn em a closed-loop transcription between a multi-class multi-dimensional data distribution and a linear discriminative representation (LDR) in the feature space. Our experiments on many benchmark imagery datasets demonstrate tremendous potential of this new closed-loop formulation.
  arXiv  Detail & Related papers  (2021-11-12T10:06:08Z)
• Implicit Greedy Rank Learning in Autoencoders via Overparameterized Linear Networks [7.412225511828064]
  Deep linear networks trained with gradient descent yield low rank solutions. We show greedy learning of low-rank latent codes induced by a linear sub-network at the autoencoder bottleneck.
  arXiv  Detail & Related papers  (2021-07-02T23:17:50Z)
• Diffusion-Based Representation Learning [65.55681678004038]
  We augment the denoising score matching framework to enable representation learning without any supervised signal. In contrast, the introduced diffusion-based representation learning relies on a new formulation of the denoising score matching objective. Using the same approach, we propose to learn an infinite-dimensional latent code that achieves improvements of state-of-the-art models on semi-supervised image classification.
  arXiv  Detail & Related papers  (2021-05-29T09:26:02Z)
• A New Modal Autoencoder for Functionally Independent Feature Extraction [6.690183908967779]
  A new modal autoencoder (MAE) is proposed by othogonalising the columns of the readout weight matrix. The results were validated on the MNIST variations and USPS classification benchmark suite. The new MAE introduces a very simple training principle for autoencoders and could be promising for the pre-training of deep neural networks.
  arXiv  Detail & Related papers  (2020-06-25T13:25:10Z)
• MetaSDF: Meta-learning Signed Distance Functions [85.81290552559817]
  Generalizing across shapes with neural implicit representations amounts to learning priors over the respective function space. We formalize learning of a shape space as a meta-learning problem and leverage gradient-based meta-learning algorithms to solve this task.
  arXiv  Detail & Related papers  (2020-06-17T05:14:53Z)
• Evolving Losses for Unsupervised Video Representation Learning [91.2683362199263]
  We present a new method to learn video representations from large-scale unlabeled video data. The proposed unsupervised representation learning results in a single RGB network and outperforms previous methods.
  arXiv  Detail & Related papers  (2020-02-26T16:56:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.