Towards Diverse, Relevant and Coherent Open-Domain Dialogue Generation via Hybrid Latent Variables

• URL: http://arxiv.org/abs/2212.01145v1
• Date: Fri, 2 Dec 2022 12:48:01 GMT
• Title: Towards Diverse, Relevant and Coherent Open-Domain Dialogue Generation via Hybrid Latent Variables
• Authors: Bin Sun, Yitong Li, Fei Mi, Weichao Wang, Yiwei Li, Kan Li
• Abstract summary: We combine the merits of both continuous and discrete latent variables and propose a Hybrid Latent Variable (HLV) method. HLV constrains the global semantics of responses through discrete latent variables and enriches responses with continuous latent variables. In addition, we propose Conditional Hybrid Variational Transformer (CHVT) to construct and to utilize HLV with transformers for dialogue generation.
• Score: 20.66743177460193
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conditional variational models, using either continuous or discrete latent variables, are powerful for open-domain dialogue response generation. However, previous works show that continuous latent variables tend to reduce the coherence of generated responses. In this paper, we also found that discrete latent variables have difficulty capturing more diverse expressions. To tackle these problems, we combine the merits of both continuous and discrete latent variables and propose a Hybrid Latent Variable (HLV) method. Specifically, HLV constrains the global semantics of responses through discrete latent variables and enriches responses with continuous latent variables. Thus, we diversify the generated responses while maintaining relevance and coherence. In addition, we propose Conditional Hybrid Variational Transformer (CHVT) to construct and to utilize HLV with transformers for dialogue generation. Through fine-grained symbolic-level semantic information and additive Gaussian mixing, we construct the distribution of continuous variables, prompting the generation of diverse expressions. Meanwhile, to maintain the relevance and coherence, the discrete latent variable is optimized by self-separation training. Experimental results on two dialogue generation datasets (DailyDialog and Opensubtitles) show that CHVT is superior to traditional transformer-based variational mechanism w.r.t. diversity, relevance and coherence metrics. Moreover, we also demonstrate the benefit of applying HLV to fine-tuning two pre-trained dialogue models (PLATO and BART-base).

Related papers

• Dior-CVAE: Pre-trained Language Models and Diffusion Priors for Variational Dialog Generation [70.2283756542824]
  Dior-CVAE is a hierarchical conditional variational autoencoder (CVAE) with diffusion priors to address these challenges. We employ a diffusion model to increase the complexity of the prior distribution and its compatibility with the distributions produced by a PLM. Experiments across two commonly used open-domain dialog datasets show that our method can generate more diverse responses without large-scale dialog pre-training.
  arXiv  Detail & Related papers  (2023-05-24T11:06:52Z)
• PVGRU: Generating Diverse and Relevant Dialogue Responses via Pseudo-Variational Mechanism [21.705686583621816]
  Existing generative models usually employ the last hidden state to summarize the sequences. We propose a Pseudo-Variational Gated Recurrent Unit (PVGRU) component without posterior knowledge. PVGRU can perceive the subtle semantic variability through summarizing variables that are optimized by the devised distribution consistency and reconstruction objectives.
  arXiv  Detail & Related papers  (2022-12-18T13:36:07Z)
• Recurrence Boosts Diversity! Revisiting Recurrent Latent Variable in Transformer-Based Variational AutoEncoder for Diverse Text Generation [85.5379146125199]
  Variational Auto-Encoder (VAE) has been widely adopted in text generation. We propose TRACE, a Transformer-based recurrent VAE structure.
  arXiv  Detail & Related papers  (2022-10-22T10:25:35Z)
• Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantization [76.68866368409216]
  We propose learning to dynamically select discretization tightness conditioned on inputs. We show that dynamically varying tightness in communication bottlenecks can improve model performance on visual reasoning and reinforcement learning tasks.
  arXiv  Detail & Related papers  (2022-02-02T23:54:26Z)
• Generating Relevant and Coherent Dialogue Responses using Self-separated Conditional Variational AutoEncoders [10.910845951559388]
  Conditional Variational AutoEncoder (CVAE) effectively increases the diversity and informativeness of responses in open-ended dialogue generation tasks. We propose Self-separated Conditional Variational AutoEncoder (abbreviated as SepaCVAE) that introduces group information to regularize the latent variables. SepaCVAE actively divides the input data into groups, and then widens the absolute difference between data pairs from distinct groups.
  arXiv  Detail & Related papers  (2021-06-07T08:19:31Z)
• Hierarchical Multi-Grained Generative Model for Expressive Speech Synthesis [19.386519810463003]
  This paper proposes a hierarchical generative model with a multi-grained latent variable to synthesize expressive speech. Our proposed framework also provides the controllability of speaking style in an entire utterance.
  arXiv  Detail & Related papers  (2020-09-17T18:00:19Z)
• Learning Disentangled Representations with Latent Variation Predictability [102.4163768995288]
  This paper defines the variation predictability of latent disentangled representations. Within an adversarial generation process, we encourage variation predictability by maximizing the mutual information between latent variations and corresponding image pairs. We develop an evaluation metric that does not rely on the ground-truth generative factors to measure the disentanglement of latent representations.
  arXiv  Detail & Related papers  (2020-07-25T08:54:26Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)
• Variational Transformers for Diverse Response Generation [71.53159402053392]
  Variational Transformer (VT) is a variational self-attentive feed-forward sequence model. VT combines the parallelizability and global receptive field computation of the Transformer with the variational nature of the CVAE. We explore two types of VT: 1) modeling the discourse-level diversity with a global latent variable; and 2) augmenting the Transformer decoder with a sequence of finegrained latent variables.
  arXiv  Detail & Related papers  (2020-03-28T07:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.