Unlocking Guidance for Discrete State-Space Diffusion and Flow Models

• URL: http://arxiv.org/abs/2406.01572v1
• Date: Mon, 3 Jun 2024 17:51:54 GMT
• Title: Unlocking Guidance for Discrete State-Space Diffusion and Flow Models
• Authors: Hunter Nisonoff, Junhao Xiong, Stephan Allenspach, Jennifer Listgarten,
• Abstract summary: We introduce a general and principled method for applying guidance on discrete state-space models. We demonstrate the utility of our approach on a range of applications including guided generation of images, small-molecules, DNA sequences and protein sequences.
• Score: 1.7749342709605143
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative models on discrete state-spaces have a wide range of potential applications, particularly in the domain of natural sciences. In continuous state-spaces, controllable and flexible generation of samples with desired properties has been realized using guidance on diffusion and flow models. However, these guidance approaches are not readily amenable to discrete state-space models. Consequently, we introduce a general and principled method for applying guidance on such models. Our method depends on leveraging continuous-time Markov processes on discrete state-spaces, which unlocks computational tractability for sampling from a desired guided distribution. We demonstrate the utility of our approach, Discrete Guidance, on a range of applications including guided generation of images, small-molecules, DNA sequences and protein sequences.

Related papers

• Constrained Discrete Diffusion [61.81569616239755]
  This paper introduces Constrained Discrete Diffusion (CDD), a novel integration of differentiable constraint optimization within the diffusion process.<n>CDD directly imposes constraints into the discrete diffusion sampling process, resulting in a training-free and effective approach.
  arXiv  Detail & Related papers  (2025-03-12T19:48:12Z)
• Continuous Diffusion Model for Language Modeling [57.396578974401734]
  Existing continuous diffusion models for discrete data have limited performance compared to discrete approaches. We propose a continuous diffusion model for language modeling that incorporates the geometry of the underlying categorical distribution.
  arXiv  Detail & Related papers  (2025-02-17T08:54:29Z)
• TFG-Flow: Training-free Guidance in Multimodal Generative Flow [73.93071065307782]
  We introduce TFG-Flow, a training-free guidance method for multimodal generative flow. TFG-Flow addresses the curse-of-dimensionality while maintaining the property of unbiased sampling in guiding discrete variables. We show that TFG-Flow has great potential in drug design by generating molecules with desired properties.
  arXiv  Detail & Related papers  (2025-01-24T03:44:16Z)
• Simple Guidance Mechanisms for Discrete Diffusion Models [44.377206440698586]
  We develop a new class of diffusion models that leverage uniform noise and that are more guidable because they can continuously edit their outputs. We improve the quality of these models with a novel continuous-time variational lower bound that yields state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-13T15:08:30Z)
• Convergence of Score-Based Discrete Diffusion Models: A Discrete-Time Analysis [56.442307356162864]
  We study the theoretical aspects of score-based discrete diffusion models under the Continuous Time Markov Chain (CTMC) framework. We introduce a discrete-time sampling algorithm in the general state space $[S]d$ that utilizes score estimators at predefined time points. Our convergence analysis employs a Girsanov-based method and establishes key properties of the discrete score function.
  arXiv  Detail & Related papers  (2024-10-03T09:07:13Z)
• Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-Based Decoding [84.3224556294803]
  Diffusion models excel at capturing the natural design spaces of images, molecules, DNA, RNA, and protein sequences. We aim to optimize downstream reward functions while preserving the naturalness of these design spaces. Our algorithm integrates soft value functions, which looks ahead to how intermediate noisy states lead to high rewards in the future.
  arXiv  Detail & Related papers  (2024-08-15T16:47:59Z)
• Context-Guided Diffusion for Out-of-Distribution Molecular and Protein Design [30.241533997522236]
  We develop context-guided diffusion (CGD), a simple plug-and-play method that leverages unlabeled data and smoothness constraints to improve the out-of-distribution generalization of guided diffusion models. This approach leads to substantial performance gains across various settings, including continuous, discrete, and graph-structured diffusion processes with applications across drug discovery, materials science, and protein design.
  arXiv  Detail & Related papers  (2024-07-16T17:34:00Z)
• Synthetic location trajectory generation using categorical diffusion models [50.809683239937584]
  Diffusion models (DPMs) have rapidly evolved to be one of the predominant generative models for the simulation of synthetic data. We propose using DPMs for the generation of synthetic individual location trajectories (ILTs) which are sequences of variables representing physical locations visited by individuals.
  arXiv  Detail & Related papers  (2024-02-19T15:57:39Z)
• Convergence Analysis of Discrete Diffusion Model: Exact Implementation through Uniformization [17.535229185525353]
  We introduce an algorithm leveraging the uniformization of continuous Markov chains, implementing transitions on random time points. Our results align with state-of-the-art achievements for diffusion models in $mathbbRd$ and further underscore the advantages of discrete diffusion models in comparison to the $mathbbRd$ setting.
  arXiv  Detail & Related papers  (2024-02-12T22:26:52Z)
• Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design [37.634098563033795]
  We present a new flow-based model of discrete data that provides the missing link in enabling flow-based generative models. Our key insight is that the discrete equivalent of continuous space flow matching can be realized using Continuous Time Markov Chains. We apply this capability to the task of protein co-design, wherein we learn a model for jointly generating protein structure and sequence.
  arXiv  Detail & Related papers  (2024-02-07T16:15:36Z)
• Fast Sampling via Discrete Non-Markov Diffusion Models [49.598085130313514]
  We propose a discrete non-Markov diffusion model, which admits an accelerated reverse sampling for discrete data generation. Our method significantly reduces the number of function evaluations (i.e., calls to the neural network), making the sampling process much faster.
  arXiv  Detail & Related papers  (2023-12-14T18:14:11Z)
• Generative Modeling on Manifolds Through Mixture of Riemannian Diffusion Processes [57.396578974401734]
  We introduce a principled framework for building a generative diffusion process on general manifold. Instead of following the denoising approach of previous diffusion models, we construct a diffusion process using a mixture of bridge processes. We develop a geometric understanding of the mixture process, deriving the drift as a weighted mean of tangent directions to the data points.
  arXiv  Detail & Related papers  (2023-10-11T06:04:40Z)
• Free-Form Variational Inference for Gaussian Process State-Space Models [21.644570034208506]
  We propose a new method for inference in Bayesian GPSSMs. Our method is based on freeform variational inference via inducing Hamiltonian Monte Carlo. We show that our approach can learn transition dynamics and latent states more accurately than competing methods.
  arXiv  Detail & Related papers  (2023-02-20T11:34:16Z)
• Diffusion Generative Models in Infinite Dimensions [10.15736468214228]
  We generalize diffusion generative models to operate directly in function space. A significant benefit of our function space point of view is that it allows us to explicitly specify the space of functions we are working in. Our approach allows us to perform both unconditional and conditional generation of function-valued data.
  arXiv  Detail & Related papers  (2022-12-01T21:54:19Z)
• A Survey on Generative Diffusion Model [75.93774014861978]
  Diffusion models are an emerging class of deep generative models. They have certain limitations, including a time-consuming iterative generation process and confinement to high-dimensional Euclidean space. This survey presents a plethora of advanced techniques aimed at enhancing diffusion models.
  arXiv  Detail & Related papers  (2022-09-06T16:56:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.