Improving Discrete Diffusion Models via Structured Preferential Generation

• URL: http://arxiv.org/abs/2405.17889v1
• Date: Tue, 28 May 2024 07:11:30 GMT
• Title: Improving Discrete Diffusion Models via Structured Preferential Generation
• Authors: Severi Rissanen, Markus Heinonen, Arno Solin,
• Abstract summary: This paper tackles the challenge of improving discrete diffusion models by introducing a structured forward process. Our approach biases the generative process to produce certain categories before others, resulting in a notable improvement in log-likelihood scores on the text8 dataset.
• Score: 25.784316302130875
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In the domains of image and audio, diffusion models have shown impressive performance. However, their application to discrete data types, such as language, has often been suboptimal compared to autoregressive generative models. This paper tackles the challenge of improving discrete diffusion models by introducing a structured forward process that leverages the inherent information hierarchy in discrete categories, such as words in text. Our approach biases the generative process to produce certain categories before others, resulting in a notable improvement in log-likelihood scores on the text8 dataset. This work paves the way for more advances in discrete diffusion models with potentially significant enhancements in performance.

Related papers

• GASE: Generatively Augmented Sentence Encoding [0.0]
  We propose an approach to enhance sentence embeddings by applying generative text models for data augmentation at inference time. Generatively Augmented Sentence uses diverse synthetic variants of input texts generated by paraphrasing, summarising or extracting keywords. We find that generative augmentation leads to larger performance improvements for embedding models with lower baseline performance.
  arXiv  Detail & Related papers  (2024-11-07T17:53:47Z)
• Energy-Based Diffusion Language Models for Text Generation [126.23425882687195]
  Energy-based Diffusion Language Model (EDLM) is an energy-based model operating at the full sequence level for each diffusion step. Our framework offers a 1.3$times$ sampling speedup over existing diffusion models.
  arXiv  Detail & Related papers  (2024-10-28T17:25:56Z)
• DetDiffusion: Synergizing Generative and Perceptive Models for Enhanced Data Generation and Perception [78.26734070960886]
  Current perceptive models heavily depend on resource-intensive datasets. We introduce perception-aware loss (P.A. loss) through segmentation, improving both quality and controllability. Our method customizes data augmentation by extracting and utilizing perception-aware attribute (P.A. Attr) during generation.
  arXiv  Detail & Related papers  (2024-03-20T04:58:03Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)
• Guided Flows for Generative Modeling and Decision Making [55.42634941614435]
  We show that Guided Flows significantly improves the sample quality in conditional image generation and zero-shot text synthesis-to-speech. Notably, we are first to apply flow models for plan generation in the offline reinforcement learning setting ax speedup in compared to diffusion models.
  arXiv  Detail & Related papers  (2023-11-22T15:07:59Z)
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution [67.9215891673174]
  We propose score entropy as a novel loss that naturally extends score matching to discrete spaces. We test our Score Entropy Discrete Diffusion models on standard language modeling tasks.
  arXiv  Detail & Related papers  (2023-10-25T17:59:12Z)
• Image retrieval outperforms diffusion models on data augmentation [36.559967424331695]
  diffusion models are proposed to augment training datasets for downstream tasks, such as classification. It remains unclear if they generalize enough to improve over directly using the additional data of their pre-training process for augmentation. Personalizing diffusion models towards the target data outperforms simpler prompting strategies. However, using the pre-training data of the diffusion model alone, via a simple nearest-neighbor retrieval procedure, leads to even stronger downstream performance.
  arXiv  Detail & Related papers  (2023-04-20T12:21:30Z)
• Diffusing Gaussian Mixtures for Generating Categorical Data [21.43283907118157]
  We propose a generative model for categorical data based on diffusion models with a focus on high-quality sample generation. Our method of evaluation highlights the capabilities and limitations of different generative models for generating categorical data.
  arXiv  Detail & Related papers  (2023-03-08T14:55:32Z)
• A Reparameterized Discrete Diffusion Model for Text Generation [39.0145272152805]
  This work studies discrete diffusion probabilistic models with applications to natural language generation. We derive an alternative yet equivalent formulation of the sampling from discrete diffusion processes. We conduct extensive experiments to evaluate the text generation capability of our model, demonstrating significant improvements over existing diffusion models.
  arXiv  Detail & Related papers  (2023-02-11T16:26:57Z)
• A Survey on Generative Diffusion Model [75.93774014861978]
  Diffusion models are an emerging class of deep generative models. They have certain limitations, including a time-consuming iterative generation process and confinement to high-dimensional Euclidean space. This survey presents a plethora of advanced techniques aimed at enhancing diffusion models.
  arXiv  Detail & Related papers  (2022-09-06T16:56:21Z)
• Diffusion Models: A Comprehensive Survey of Methods and Applications [10.557289965753437]
  Diffusion models are a class of deep generative models that have shown impressive results on various tasks with dense theoretical founding. Recent studies have shown great enthusiasm on improving the performance of diffusion model.
  arXiv  Detail & Related papers  (2022-09-02T02:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.