A Sparsity-promoting Dictionary Model for Variational Autoencoders
- URL: http://arxiv.org/abs/2203.15758v1
- Date: Tue, 29 Mar 2022 17:13:11 GMT
- Title: A Sparsity-promoting Dictionary Model for Variational Autoencoders
- Authors: Mostafa Sadeghi, Paul Magron
- Abstract summary: Structuring the latent space in deep generative models is important to yield more expressive models and interpretable representations.
We propose a simple yet effective methodology to structure the latent space via a sparsity-promoting dictionary model.
- Score: 16.61511959679188
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Structuring the latent space in probabilistic deep generative models, e.g.,
variational autoencoders (VAEs), is important to yield more expressive models
and interpretable representations, and to avoid overfitting. One way to achieve
this objective is to impose a sparsity constraint on the latent variables,
e.g., via a Laplace prior. However, such approaches usually complicate the
training phase, and they sacrifice the reconstruction quality to promote
sparsity. In this paper, we propose a simple yet effective methodology to
structure the latent space via a sparsity-promoting dictionary model, which
assumes that each latent code can be written as a sparse linear combination of
a dictionary's columns. In particular, we leverage a computationally efficient
and tuning-free method, which relies on a zero-mean Gaussian latent prior with
learnable variances. We derive a variational inference scheme to train the
model. Experiments on speech generative modeling demonstrate the advantage of
the proposed approach over competing techniques, since it promotes sparsity
while not deteriorating the output speech quality.
Related papers
- Faster Language Models with Better Multi-Token Prediction Using Tensor Decomposition [5.575078692353885]
We propose a new model for multi-token prediction in transformers, aiming to enhance sampling efficiency without compromising accuracy.
By generalizing it to a rank-$r$ canonical probability decomposition, we develop an improved model that predicts multiple tokens simultaneously.
arXiv Detail & Related papers (2024-10-23T11:06:36Z) - Disentanglement with Factor Quantized Variational Autoencoders [11.086500036180222]
We propose a discrete variational autoencoder (VAE) based model where the ground truth information about the generative factors are not provided to the model.
We demonstrate the advantages of learning discrete representations over learning continuous representations in facilitating disentanglement.
Our method called FactorQVAE is the first method that combines optimization based disentanglement approaches with discrete representation learning.
arXiv Detail & Related papers (2024-09-23T09:33:53Z) - Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding [84.3224556294803]
Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences.
We aim to optimize downstream reward functions while preserving the naturalness of these design spaces.
Our algorithm integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future.
arXiv Detail & Related papers (2024-08-15T16:47:59Z) - Disentanglement via Latent Quantization [60.37109712033694]
In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space.
We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
arXiv Detail & Related papers (2023-05-28T06:30:29Z) - A weighted-variance variational autoencoder model for speech enhancement [0.0]
We propose a weighted variance generative model, where the contribution of each spectrogram time-frame in parameter learning is weighted.
We develop efficient training and speech enhancement algorithms based on the proposed generative model.
arXiv Detail & Related papers (2022-11-02T09:51:15Z) - Structured Reordering for Modeling Latent Alignments in Sequence
Transduction [86.94309120789396]
We present an efficient dynamic programming algorithm performing exact marginal inference of separable permutations.
The resulting seq2seq model exhibits better systematic generalization than standard models on synthetic problems and NLP tasks.
arXiv Detail & Related papers (2021-06-06T21:53:54Z) - Generative Text Modeling through Short Run Inference [47.73892773331617]
The present work proposes a short run dynamics for inference. It is variation from the prior distribution of the latent variable and then runs a small number of Langevin dynamics steps guided by its posterior distribution.
We show that the models trained with short run dynamics more accurately model the data, compared to strong language model and VAE baselines, and exhibit no sign of posterior collapse.
arXiv Detail & Related papers (2021-05-27T09:14:35Z) - Preventing Posterior Collapse with Levenshtein Variational Autoencoder [61.30283661804425]
We propose to replace the evidence lower bound (ELBO) with a new objective which is simple to optimize and prevents posterior collapse.
We show that Levenstein VAE produces more informative latent representations than alternative approaches to preventing posterior collapse.
arXiv Detail & Related papers (2020-04-30T13:27:26Z) - Improve Variational Autoencoder for Text Generationwith Discrete Latent
Bottleneck [52.08901549360262]
Variational autoencoders (VAEs) are essential tools in end-to-end representation learning.
VAEs tend to ignore latent variables with a strong auto-regressive decoder.
We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
arXiv Detail & Related papers (2020-04-22T14:41:37Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.