A Sparsity-promoting Dictionary Model for Variational Autoencoders
- URL: http://arxiv.org/abs/2203.15758v1
- Date: Tue, 29 Mar 2022 17:13:11 GMT
- Title: A Sparsity-promoting Dictionary Model for Variational Autoencoders
- Authors: Mostafa Sadeghi, Paul Magron
- Abstract summary: Structuring the latent space in deep generative models is important to yield more expressive models and interpretable representations.
We propose a simple yet effective methodology to structure the latent space via a sparsity-promoting dictionary model.
- Score: 16.61511959679188
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Structuring the latent space in probabilistic deep generative models, e.g.,
variational autoencoders (VAEs), is important to yield more expressive models
and interpretable representations, and to avoid overfitting. One way to achieve
this objective is to impose a sparsity constraint on the latent variables,
e.g., via a Laplace prior. However, such approaches usually complicate the
training phase, and they sacrifice the reconstruction quality to promote
sparsity. In this paper, we propose a simple yet effective methodology to
structure the latent space via a sparsity-promoting dictionary model, which
assumes that each latent code can be written as a sparse linear combination of
a dictionary's columns. In particular, we leverage a computationally efficient
and tuning-free method, which relies on a zero-mean Gaussian latent prior with
learnable variances. We derive a variational inference scheme to train the
model. Experiments on speech generative modeling demonstrate the advantage of
the proposed approach over competing techniques, since it promotes sparsity
while not deteriorating the output speech quality.
Related papers
- Disentanglement via Latent Quantization [60.37109712033694]
In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space.
We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
arXiv Detail & Related papers (2023-05-28T06:30:29Z) - A weighted-variance variational autoencoder model for speech enhancement [0.0]
We propose a weighted variance generative model, where the contribution of each spectrogram time-frame in parameter learning is weighted.
We develop efficient training and speech enhancement algorithms based on the proposed generative model.
arXiv Detail & Related papers (2022-11-02T09:51:15Z) - Structured Reordering for Modeling Latent Alignments in Sequence
Transduction [86.94309120789396]
We present an efficient dynamic programming algorithm performing exact marginal inference of separable permutations.
The resulting seq2seq model exhibits better systematic generalization than standard models on synthetic problems and NLP tasks.
arXiv Detail & Related papers (2021-06-06T21:53:54Z) - Generative Text Modeling through Short Run Inference [47.73892773331617]
The present work proposes a short run dynamics for inference. It is variation from the prior distribution of the latent variable and then runs a small number of Langevin dynamics steps guided by its posterior distribution.
We show that the models trained with short run dynamics more accurately model the data, compared to strong language model and VAE baselines, and exhibit no sign of posterior collapse.
arXiv Detail & Related papers (2021-05-27T09:14:35Z) - Generative Model without Prior Distribution Matching [26.91643368299913]
Variational Autoencoder (VAE) and its variations are classic generative models by learning a low-dimensional latent representation to satisfy some prior distribution.
We propose to let the prior match the embedding distribution rather than imposing the latent variables to fit the prior.
arXiv Detail & Related papers (2020-09-23T09:33:24Z) - Generative Semantic Hashing Enhanced via Boltzmann Machines [61.688380278649056]
Existing generative-hashing methods mostly assume a factorized form for the posterior distribution.
We propose to employ the distribution of Boltzmann machine as the retrievalal posterior.
We show that by effectively modeling correlations among different bits within a hash code, our model can achieve significant performance gains.
arXiv Detail & Related papers (2020-06-16T01:23:39Z) - Preventing Posterior Collapse with Levenshtein Variational Autoencoder [61.30283661804425]
We propose to replace the evidence lower bound (ELBO) with a new objective which is simple to optimize and prevents posterior collapse.
We show that Levenstein VAE produces more informative latent representations than alternative approaches to preventing posterior collapse.
arXiv Detail & Related papers (2020-04-30T13:27:26Z) - Improve Variational Autoencoder for Text Generationwith Discrete Latent
Bottleneck [52.08901549360262]
Variational autoencoders (VAEs) are essential tools in end-to-end representation learning.
VAEs tend to ignore latent variables with a strong auto-regressive decoder.
We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
arXiv Detail & Related papers (2020-04-22T14:41:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.