Deep Autoencoders: From Understanding to Generalization Guarantees
- URL: http://arxiv.org/abs/2009.09525v3
- Date: Wed, 24 Nov 2021 18:43:46 GMT
- Title: Deep Autoencoders: From Understanding to Generalization Guarantees
- Authors: Romain Cosentino, Randall Balestriero, Richard Baraniuk, Behnaam
Aazhang
- Abstract summary: We take a step towards a better understanding of the underlying phenomena of Deep Autoencoders (AEs)
In particular, we interpret how AEs approximate the data manifold by exploiting their continuous piecewise affine structure.
We derive two new regularizations that enable AEs to capture the inherent symmetry in the data.
- Score: 17.180863761413004
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: A big mystery in deep learning continues to be the ability of methods to
generalize when the number of model parameters is larger than the number of
training examples. In this work, we take a step towards a better understanding
of the underlying phenomena of Deep Autoencoders (AEs), a mainstream deep
learning solution for learning compressed, interpretable, and structured data
representations. In particular, we interpret how AEs approximate the data
manifold by exploiting their continuous piecewise affine structure. Our
reformulation of AEs provides new insights into their mapping, reconstruction
guarantees, as well as an interpretation of commonly used regularization
techniques. We leverage these findings to derive two new regularizations that
enable AEs to capture the inherent symmetry in the data. Our regularizations
leverage recent advances in the group of transformation learning to enable AEs
to better approximate the data manifold without explicitly defining the group
underlying the manifold. Under the assumption that the symmetry of the data can
be explained by a Lie group, we prove that the regularizations ensure the
generalization of the corresponding AEs. A range of experimental evaluations
demonstrate that our methods outperform other state-of-the-art regularization
techniques.
Related papers
- Effort: Efficient Orthogonal Modeling for Generalizable AI-Generated Image Detection [66.16595174895802]
Existing AI-generated image (AIGI) detection methods often suffer from limited generalization performance.
In this paper, we identify a crucial yet previously overlooked asymmetry phenomenon in AIGI detection.
arXiv Detail & Related papers (2024-11-23T19:10:32Z) - Slicing Mutual Information Generalization Bounds for Neural Networks [14.48773730230054]
We introduce new, tighter information-theoretic generalization bounds tailored for deep learning algorithms.
Our bounds offer significant computational and statistical advantages over standard MI bounds.
We extend our analysis to algorithms whose parameters do not need to exactly lie on random subspaces.
arXiv Detail & Related papers (2024-06-06T13:15:37Z) - Disentanglement via Latent Quantization [60.37109712033694]
In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space.
We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
arXiv Detail & Related papers (2023-05-28T06:30:29Z) - Revisiting Structured Variational Autoencoders [11.998116457994994]
Structured variational autoencoders (SVAEs) combine probabilistic graphical model priors on latent variables, deep neural networks to link latent variables to observed data, and structure-exploiting algorithms for approximate posterior inference.
Despite their conceptual elegance, SVAEs have proven difficult to implement, and more general approaches have been favored in practice.
Here, we revisit SVAEs using modern machine learning tools and demonstrate their advantages over more general alternatives in terms of both accuracy and efficiency.
arXiv Detail & Related papers (2023-05-25T23:51:18Z) - DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained
Diffusion [66.21290235237808]
We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states.
We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs.
Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
arXiv Detail & Related papers (2023-01-23T15:18:54Z) - Improved Representation Learning Through Tensorized Autoencoders [7.056005298953332]
Autoencoders (AE) are widely used in practice for unsupervised representation learning.
We propose a meta-algorithm that can be used to extend an arbitrary AE architecture to a tensorized version (TAE)
We prove that TAE can recover the principle components of the different clusters in contrast to principle component of the entire data recovered by a standard AE.
arXiv Detail & Related papers (2022-12-02T09:29:48Z) - ER: Equivariance Regularizer for Knowledge Graph Completion [107.51609402963072]
We propose a new regularizer, namely, Equivariance Regularizer (ER)
ER can enhance the generalization ability of the model by employing the semantic equivariance between the head and tail entities.
The experimental results indicate a clear and substantial improvement over the state-of-the-art relation prediction methods.
arXiv Detail & Related papers (2022-06-24T08:18:05Z) - Toward Learning Robust and Invariant Representations with Alignment
Regularization and Data Augmentation [76.85274970052762]
This paper is motivated by a proliferation of options of alignment regularizations.
We evaluate the performances of several popular design choices along the dimensions of robustness and invariance.
We also formally analyze the behavior of alignment regularization to complement our empirical study under assumptions we consider realistic.
arXiv Detail & Related papers (2022-06-04T04:29:19Z) - Regularizing Variational Autoencoder with Diversity and Uncertainty
Awareness [61.827054365139645]
Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference.
We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
arXiv Detail & Related papers (2021-10-24T07:58:13Z) - Extendable and invertible manifold learning with geometry regularized
autoencoders [9.742277703732187]
A fundamental task in data exploration is to extract simplified low dimensional representations that capture intrinsic geometry in data.
Common approaches to this task use kernel methods for manifold learning.
We present a new method for integrating both approaches by incorporating a geometric regularization term in the bottleneck of the autoencoder.
arXiv Detail & Related papers (2020-07-14T15:59:10Z) - ARAE: Adversarially Robust Training of Autoencoders Improves Novelty
Detection [6.992807725367106]
Autoencoders (AE) have been widely employed to approach the novelty detection problem.
We propose a novel AE that can learn more semantically meaningful features.
We show that despite using a much simpler architecture, the proposed AE outperforms or is competitive to state-of-the-art on three benchmark datasets.
arXiv Detail & Related papers (2020-03-12T09:06:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.