VAE Approximation Error: ELBO and Conditional Independence
- URL: http://arxiv.org/abs/2102.09310v1
- Date: Thu, 18 Feb 2021 12:54:42 GMT
- Title: VAE Approximation Error: ELBO and Conditional Independence
- Authors: Dmitrij Schlesinger, Alexander Shekhovtsov, Boris Flach
- Abstract summary: This paper analyzes VAE approximation errors caused by the combination of the ELBO objective with the choice of the encoder probability family.
We show that the ELBO subset can not be enlarged, and the respective error cannot be decreased, by only considering deeper encoder networks.
- Score: 78.72292013299868
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: The importance of Variational Autoencoders reaches far beyond standalone
generative models -- the approach is also used for learning latent
representations and can be generalized to semi-supervised learning. This
requires a thorough analysis of their commonly known shortcomings: posterior
collapse and approximation errors. This paper analyzes VAE approximation errors
caused by the combination of the ELBO objective with the choice of the encoder
probability family, in particular under conditional independence assumptions.
We identify the subclass of generative models consistent with the encoder
family. We show that the ELBO optimizer is pulled from the likelihood optimizer
towards this consistent subset. Furthermore, this subset can not be enlarged,
and the respective error cannot be decreased, by only considering deeper
encoder networks.
Related papers
- Likelihood-based Differentiable Structure Learning [38.25218464260965]
Existing approaches to differentiable structure learning of directed acyclic graphs (DAGs) rely on strong identifiability assumptions.
We explain and remedy these issues by studying the behavior of differentiable acyclicity-constrained programs under general likelihoods.
arXiv Detail & Related papers (2024-10-08T16:08:24Z) - Fusing Dictionary Learning and Support Vector Machines for Unsupervised Anomaly Detection [1.5999407512883508]
We introduce a new anomaly detection model that unifies the OC-SVM and DL residual functions into a single composite objective.
We extend both objectives to the more general setting that allows the use of kernel functions.
arXiv Detail & Related papers (2024-04-05T12:41:53Z) - On the failure of variational score matching for VAE models [3.8073142980733]
We present a critical study of existing variational SM objectives, showing catastrophic failure on a wide range of datasets and network architectures.
Our theoretical insights on the objectives emerge directly from their equivalent autoencoding losses when optimizing variational autoencoder (VAE) models.
arXiv Detail & Related papers (2022-10-24T16:43:04Z) - Embrace the Gap: VAEs Perform Independent Mechanism Analysis [36.686468842036305]
We study nonlinear VAEs in the limit of near-deterministic decoders.
We show that VAEs uncover the true latent factors when the data generating process satisfies the IMA assumption.
arXiv Detail & Related papers (2022-06-06T08:19:19Z) - Predicting Unreliable Predictions by Shattering a Neural Network [145.3823991041987]
Piecewise linear neural networks can be split into subfunctions.
Subfunctions have their own activation pattern, domain, and empirical error.
Empirical error for the full network can be written as an expectation over subfunctions.
arXiv Detail & Related papers (2021-06-15T18:34:41Z) - Outlier-Robust Learning of Ising Models Under Dobrushin's Condition [57.89518300699042]
We study the problem of learning Ising models satisfying Dobrushin's condition in the outlier-robust setting where a constant fraction of the samples are adversarially corrupted.
Our main result is to provide the first computationally efficient robust learning algorithm for this problem with near-optimal error guarantees.
arXiv Detail & Related papers (2021-02-03T18:00:57Z) - Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
Variational autoencoders (VAE) are a powerful and widely-used class of generative models.
We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs.
Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
arXiv Detail & Related papers (2021-01-06T17:36:26Z) - Autoencoding Variational Autoencoder [56.05008520271406]
We study the implications of this behaviour on the learned representations and also the consequences of fixing it by introducing a notion of self consistency.
We show that encoders trained with our self-consistency approach lead to representations that are robust (insensitive) to perturbations in the input introduced by adversarial attacks.
arXiv Detail & Related papers (2020-12-07T14:16:14Z) - General stochastic separation theorems with optimal bounds [68.8204255655161]
Phenomenon of separability was revealed and used in machine learning to correct errors of Artificial Intelligence (AI) systems and analyze AI instabilities.
Errors or clusters of errors can be separated from the rest of the data.
The ability to correct an AI system also opens up the possibility of an attack on it, and the high dimensionality induces vulnerabilities caused by the same separability.
arXiv Detail & Related papers (2020-10-11T13:12:41Z) - Generalizing Variational Autoencoders with Hierarchical Empirical Bayes [6.273154057349038]
We present Hierarchical Empirical Bayes Autoencoder (HEBAE), a computationally stable framework for probabilistic generative models.
Our key contributions are two-fold. First, we make gains by placing a hierarchical prior over the encoding distribution, enabling us to adaptively balance the trade-off between minimizing the reconstruction loss function and avoiding over-regularization.
arXiv Detail & Related papers (2020-07-20T18:18:39Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.