InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation

• URL: http://arxiv.org/abs/2310.11976v1
• Date: Wed, 18 Oct 2023 14:01:39 GMT
• Title: InfoDiffusion: Information Entropy Aware Diffusion Process for Non-Autoregressive Text Generation
• Authors: Renzhi Wang, Jing Li, Piji Li
• Abstract summary: We propose InfoDiffusion, a non-autoregressive text diffusion model. Our approach introduces a "keyinfo-first" generation strategy and incorporates a noise schedule based on the amount of text information. Experimental results show that InfoDiffusion outperforms the baseline model in terms of generation quality and diversity.
• Score: 33.52794666968048
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion models have garnered considerable interest in the field of text generation. Several studies have explored text diffusion models with different structures and applied them to various tasks, including named entity recognition and summarization. However, there exists a notable disparity between the "easy-first" text generation process of current diffusion models and the "keyword-first" natural text generation process of humans, which has received limited attention. To bridge this gap, we propose InfoDiffusion, a non-autoregressive text diffusion model. Our approach introduces a "keyinfo-first" generation strategy and incorporates a noise schedule based on the amount of text information. In addition, InfoDiffusion combines self-conditioning with a newly proposed partially noising model structure. Experimental results show that InfoDiffusion outperforms the baseline model in terms of generation quality and diversity, as well as exhibiting higher sampling efficiency.

Related papers

• Table-to-Text Generation with Pretrained Diffusion Models [0.0]
  Diffusion models have demonstrated significant potential in achieving state-of-the-art performance across various text generation tasks. We investigate their application to the table-to-text problem by adapting the diffusion model to the task and conducting an in-depth analysis. Our findings reveal that diffusion models achieve comparable results in the table-to-text domain.
  arXiv  Detail & Related papers  (2024-09-10T15:36:53Z)
• Enforcing Paraphrase Generation via Controllable Latent Diffusion [60.82512050963046]
  We propose textitLatent textitDiffusion textitParaphraser(LDP), a novel paraphrase generation by modeling a controllable diffusion process. Experiments show that LDP achieves improved and diverse paraphrase generation compared to baselines.
  arXiv  Detail & Related papers  (2024-04-13T09:24:32Z)
• DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability [75.9781362556431]
  We propose DiffDis to unify the cross-modal generative and discriminative pretraining into one single framework under the diffusion process. We show that DiffDis outperforms single-task models on both the image generation and the image-text discriminative tasks.
  arXiv  Detail & Related papers  (2023-08-18T05:03:48Z)
• PLANNER: Generating Diversified Paragraph via Latent Language Diffusion Model [37.2192243883707]
  We propose PLANNER, a model that combines latent semantic diffusion with autoregressive generation to generate fluent text. Results on semantic generation, text completion and summarization show its effectiveness in generating high-quality long-form text.
  arXiv  Detail & Related papers  (2023-06-05T01:36:39Z)
• TESS: Text-to-Text Self-Conditioned Simplex Diffusion [56.881170312435444]
  Text-to-text Self-conditioned Simplex Diffusion employs a new form of self-conditioning, and applies the diffusion process on the logit simplex space rather than the learned embedding space. We demonstrate that TESS outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models.
  arXiv  Detail & Related papers  (2023-05-15T06:33:45Z)
• Diffusion Models for Non-autoregressive Text Generation: A Survey [94.4634088113513]
  Non-autoregressive (NAR) text generation has attracted much attention in the field of natural language processing. Recently, diffusion models have been introduced into NAR text generation, showing an improved text generation quality.
  arXiv  Detail & Related papers  (2023-03-12T05:11:09Z)
• SeqDiffuSeq: Text Diffusion with Encoder-Decoder Transformers [50.90457644954857]
  In this work, we apply diffusion models to approach sequence-to-sequence text generation. We propose SeqDiffuSeq, a text diffusion model for sequence-to-sequence generation. Experiment results illustrate the good performance on sequence-to-sequence generation in terms of text quality and inference time.
  arXiv  Detail & Related papers  (2022-12-20T15:16:24Z)
• DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models [81.84866217721361]
  DiffusionBERT is a new generative masked language model based on discrete diffusion models. We propose a new noise schedule for the forward diffusion process that controls the degree of noise added at each step. Experiments on unconditional text generation demonstrate that DiffusionBERT achieves significant improvement over existing diffusion models for text.
  arXiv  Detail & Related papers  (2022-11-28T03:25:49Z)
• Self-conditioned Embedding Diffusion for Text Generation [28.342735885752493]
  Self-conditioned Embedding Diffusion is a continuous diffusion mechanism that operates on token embeddings. We show that our text diffusion models generate samples comparable with those produced by standard autoregressive language models.
  arXiv  Detail & Related papers  (2022-11-08T13:30:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.