Diffusion Models for Non-autoregressive Text Generation: A Survey

• URL: http://arxiv.org/abs/2303.06574v2
• Date: Sat, 13 May 2023 12:42:49 GMT
• Title: Diffusion Models for Non-autoregressive Text Generation: A Survey
• Authors: Yifan Li, Kun Zhou, Wayne Xin Zhao, Ji-Rong Wen
• Abstract summary: Non-autoregressive (NAR) text generation has attracted much attention in the field of natural language processing. Recently, diffusion models have been introduced into NAR text generation, showing an improved text generation quality.
• Score: 94.4634088113513
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Non-autoregressive (NAR) text generation has attracted much attention in the field of natural language processing, which greatly reduces the inference latency but has to sacrifice the generation accuracy. Recently, diffusion models, a class of latent variable generative models, have been introduced into NAR text generation, showing an improved text generation quality. In this survey, we review the recent progress in diffusion models for NAR text generation. As the background, we first present the general definition of diffusion models and the text diffusion models, and then discuss their merits for NAR generation. As the core content, we further introduce two mainstream diffusion models in existing work of text diffusion, and review the key designs of the diffusion process. Moreover, we discuss the utilization of pre-trained language models (PLMs) for text diffusion models and introduce optimization techniques for text data. Finally, we discuss several promising directions and conclude this paper. Our survey aims to provide researchers with a systematic reference of related research on text diffusion models for NAR generation. We present our collection of text diffusion models at https://github.com/RUCAIBox/Awesome-Text-Diffusion-Models.

Related papers

• InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation [33.52794666968048]
  We propose InfoDiffusion, a non-autoregressive text diffusion model. Our approach introduces a "keyinfo-first" generation strategy and incorporates a noise schedule based on the amount of text information. Experimental results show that InfoDiffusion outperforms the baseline model in terms of generation quality and diversity.
  arXiv  Detail & Related papers  (2023-10-18T14:01:39Z)
• A Survey of Diffusion Models in Natural Language Processing [11.233768932957771]
  Diffusion models capture the diffusion of information or signals across a network or manifold. This paper discusses the different formulations of diffusion models used in NLP, their strengths and limitations, and their applications.
  arXiv  Detail & Related papers  (2023-05-24T03:25:32Z)
• TESS: Text-to-Text Self-Conditioned Simplex Diffusion [56.881170312435444]
  Text-to-text Self-conditioned Simplex Diffusion employs a new form of self-conditioning, and applies the diffusion process on the logit simplex space rather than the learned embedding space. We demonstrate that TESS outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models.
  arXiv  Detail & Related papers  (2023-05-15T06:33:45Z)
• A Cheaper and Better Diffusion Language Model with Soft-Masked Noise [62.719656543880596]
  Masked-Diffuse LM is a novel diffusion model for language modeling, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. We demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.
  arXiv  Detail & Related papers  (2023-04-10T17:58:42Z)
• Text-to-image Diffusion Models in Generative AI: A Survey [75.32882187215394]
  We present a review of state-of-the-art methods on text-conditioned image synthesis, i.e., text-to-image. We discuss applications beyond text-to-image generation: text-guided creative generation and text-guided image editing.
  arXiv  Detail & Related papers  (2023-03-14T13:49:54Z)
• Diffusion Models in NLP: A Survey [1.5138755188783584]
  Diffusion models have become a powerful family of deep generative models, with record-breaking performance in many applications. This paper first gives an overview and derivation of the basic theory of diffusion models, then reviews the research results of diffusion models in the field of natural language processing.
  arXiv  Detail & Related papers  (2023-03-14T01:53:49Z)
• DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models [81.84866217721361]
  DiffusionBERT is a new generative masked language model based on discrete diffusion models. We propose a new noise schedule for the forward diffusion process that controls the degree of noise added at each step. Experiments on unconditional text generation demonstrate that DiffusionBERT achieves significant improvement over existing diffusion models for text.
  arXiv  Detail & Related papers  (2022-11-28T03:25:49Z)
• Self-conditioned Embedding Diffusion for Text Generation [28.342735885752493]
  Self-conditioned Embedding Diffusion is a continuous diffusion mechanism that operates on token embeddings. We show that our text diffusion models generate samples comparable with those produced by standard autoregressive language models.
  arXiv  Detail & Related papers  (2022-11-08T13:30:27Z)
• A Survey on Generative Diffusion Model [75.93774014861978]
  Diffusion models are an emerging class of deep generative models. They have certain limitations, including a time-consuming iterative generation process and confinement to high-dimensional Euclidean space. This survey presents a plethora of advanced techniques aimed at enhancing diffusion models.
  arXiv  Detail & Related papers  (2022-09-06T16:56:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.