Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck

• URL: http://arxiv.org/abs/2004.10603v2
• Date: Thu, 25 Feb 2021 16:16:28 GMT
• Title: Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck
• Authors: Yang Zhao, Ping Yu, Suchismit Mahapatra, Qinliang Su and Changyou Chen
• Abstract summary: Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
• Score: 52.08901549360262
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. However, the sequential text generation common pitfall with VAEs is that the model tends to ignore latent variables with a strong auto-regressive decoder. In this paper, we propose a principled approach to alleviate this issue by applying a discretized bottleneck to enforce an implicit latent feature matching in a more compact latent space. We impose a shared discrete latent space where each input is learned to choose a combination of latent atoms as a regularized latent representation. Our model endows a promising capability to model underlying semantics of discrete sequences and thus provide more interpretative latent structures. Empirically, we demonstrate our model's efficiency and effectiveness on a broad range of tasks, including language modeling, unaligned text style transfer, dialog response generation, and neural machine translation.

Related papers

• Autoregressive Speech Synthesis without Vector Quantization [135.4776759536272]
  We present MELLE, a novel continuous-valued tokens based language modeling approach for text to speech synthesis (TTS) MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition.
  arXiv  Detail & Related papers  (2024-07-11T14:36:53Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• Learning Semantic Textual Similarity via Topic-informed Discrete Latent Variables [17.57873577962635]
  We develop a topic-informed discrete latent variable model for semantic textual similarity. Our model learns a shared latent space for sentence-pair representation via vector quantization. We show that our model is able to surpass several strong neural baselines in semantic textual similarity tasks.
  arXiv  Detail & Related papers  (2022-11-07T15:09:58Z)
• Dynamic Latent Separation for Deep Learning [67.62190501599176]
  A core problem in machine learning is to learn expressive latent variables for model prediction on complex data. Here, we develop an approach that improves expressiveness, provides partial interpretation, and is not restricted to specific applications.
  arXiv  Detail & Related papers  (2022-10-07T17:56:53Z)
• A Sparsity-promoting Dictionary Model for Variational Autoencoders [16.61511959679188]
  Structuring the latent space in deep generative models is important to yield more expressive models and interpretable representations. We propose a simple yet effective methodology to structure the latent space via a sparsity-promoting dictionary model.
  arXiv  Detail & Related papers  (2022-03-29T17:13:11Z)
• Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization [76.68866368409216]
  We propose learning to dynamically select discretization tightness conditioned on inputs. We show that dynamically varying tightness in communication bottlenecks can improve model performance on visual reasoning and reinforcement learning tasks.
  arXiv  Detail & Related papers  (2022-02-02T23:54:26Z)
• Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders [0.0]
  We argue that continuous variables may not be ideal to model features of textual data, due to the fact that most generative factors in text are discrete. We propose a Variational Autoencoder based method which models language features as discrete variables and encourages independence between variables for learning disentangled representations.
  arXiv  Detail & Related papers  (2021-09-15T09:10:05Z)
• Discrete Auto-regressive Variational Attention Models for Text Modeling [53.38382932162732]
  Variational autoencoders (VAEs) have been widely applied for text modeling. They are troubled by two challenges: information underrepresentation and posterior collapse. We propose Discrete Auto-regressive Variational Attention Model (DAVAM) to address the challenges.
  arXiv  Detail & Related papers  (2021-06-16T06:36:26Z)
• Robust Training of Vector Quantized Bottleneck Models [21.540133031071438]
  We demonstrate methods for reliable and efficient training of discrete representation using Vector-Quantized Variational Auto-Encoder models (VQ-VAEs) For unsupervised representation learning, they became viable alternatives to continuous latent variable models such as the Variational Auto-Encoder (VAE)
  arXiv  Detail & Related papers  (2020-05-18T08:23:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.