DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer

• URL: http://arxiv.org/abs/2110.05999v1
• Date: Tue, 12 Oct 2021 13:41:06 GMT
• Title: DiscoDVT: Generating Long Text with Discourse-Aware Discrete Variational Transformer
• Authors: Haozhe Ji, Minlie Huang
• Abstract summary: We propose DiscoDVT, a discourse-aware discrete variational Transformer to tackle the incoherence issue. We conduct extensive experiments on two open story generation datasets and demonstrate that the latent codes learn meaningful correspondence to the discourse structures that guide the model to generate long texts with better long-range coherence.
• Score: 40.10695204278747
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the recent advances in applying pre-trained language models to generate high-quality texts, generating long passages that maintain long-range coherence is yet challenging for these models. In this paper, we propose DiscoDVT, a discourse-aware discrete variational Transformer to tackle the incoherence issue. DiscoDVT learns a discrete variable sequence that summarizes the global structure of the text and then applies it to guide the generation process at each decoding step. To further embed discourse-aware information into the discrete latent representations, we introduce an auxiliary objective to model the discourse relations within the text. We conduct extensive experiments on two open story generation datasets and demonstrate that the latent codes learn meaningful correspondence to the discourse structures that guide the model to generate long texts with better long-range coherence.

Related papers

• Vector-Quantized Prompt Learning for Paraphrase Generation [18.40940464497253]
  This paper proposes to generate diverse and high-quality paraphrases by exploiting the pre-trained models with instance-dependent prompts. Extensive experiments demonstrate that the proposed method achieves new state-of-art results on three benchmark datasets.
  arXiv  Detail & Related papers  (2023-11-25T07:13:06Z)
• Generating Coherent Narratives by Learning Dynamic and Discrete Entity States with a Contrastive Framework [68.1678127433077]
  We extend the Transformer model to dynamically conduct entity state updates and sentence realization for narrative generation. Experiments on two narrative datasets show that our model can generate more coherent and diverse narratives than strong baselines.
  arXiv  Detail & Related papers  (2022-08-08T09:02:19Z)
• PLANET: Dynamic Content Planning in Autoregressive Transformers for Long-form Text Generation [47.97523895218194]
  We propose a novel generation framework leveraging autoregressive self-attention mechanism to conduct content planning and surface realization dynamically. Our framework enriches the Transformer decoder with latent representations to maintain sentence-level semantic plans grounded by bag-of-words.
  arXiv  Detail & Related papers  (2022-03-17T05:52:35Z)
• Data-to-text Generation with Variational Sequential Planning [74.3955521225497]
  We consider the task of data-to-text generation, which aims to create textual output from non-linguistic input. We propose a neural model enhanced with a planning component responsible for organizing high-level information in a coherent and meaningful way. We infer latent plans sequentially with a structured variational model, while interleaving the steps of planning and generation.
  arXiv  Detail & Related papers  (2022-02-28T13:17:59Z)
• Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence [59.51720326054546]
  We propose a long text generation model, which can represent the prefix sentences at sentence level and discourse level in the decoding process. Our model can generate more coherent texts than state-of-the-art baselines.
  arXiv  Detail & Related papers  (2021-05-19T07:29:08Z)
• Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck [52.08901549360262]
  Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. VAEs tend to ignore latent variables with a strong auto-regressive decoder. We propose a principled approach to enforce an implicit latent feature matching in a more compact latent space.
  arXiv  Detail & Related papers  (2020-04-22T14:41:37Z)
• Syntax-driven Iterative Expansion Language Models for Controllable Text Generation [2.578242050187029]
  We propose a new paradigm for introducing a syntactic inductive bias into neural text generation. Our experiments show that this paradigm is effective at text generation, with quality between LSTMs and Transformers, and comparable diversity.
  arXiv  Detail & Related papers  (2020-04-05T14:29:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.