Constrained Discrete Diffusion

• URL: http://arxiv.org/abs/2503.09790v2
• Date: Tue, 27 May 2025 23:48:45 GMT
• Title: Constrained Discrete Diffusion
• Authors: Michael Cardei, Jacob K Christopher, Thomas Hartvigsen, Brian R. Bartoldson, Bhavya Kailkhura, Ferdinando Fioretto,
• Abstract summary: This paper introduces Constrained Discrete Diffusion (CDD), a novel integration of differentiable constraint optimization within the diffusion process.<n>CDD directly imposes constraints into the discrete diffusion sampling process, resulting in a training-free and effective approach.
• Score: 61.81569616239755
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Discrete diffusion models are a class of generative models that construct sequences by progressively denoising samples from a categorical noise distribution. Beyond their rapidly growing ability to generate coherent natural language, these models present a new and important opportunity to enforce sequence-level constraints, a capability that current autoregressive models cannot natively provide. This paper capitalizes on this opportunity by introducing Constrained Discrete Diffusion (CDD), a novel integration of differentiable constraint optimization within the diffusion process to ensure adherence to constraints, logic rules, or safety requirements for generated sequences. Unlike conventional text generators that often rely on post-hoc filtering or model retraining for controllable generation, CDD directly imposes constraints into the discrete diffusion sampling process, resulting in a training-free and effective approach. Experiments in toxicity-controlled text generation, property-constrained molecule design, and instruction-constrained text completion demonstrate that CDD achieves zero constraint violations in a diverse array of tasks while preserving fluency, novelty, and coherence while outperforming autoregressive and existing discrete diffusion approaches.

Related papers

• Unifying Continuous and Discrete Text Diffusion with Non-simultaneous Diffusion Processes [9.29387855908007]
  NeoDiff is a novel diffusion model that integrates the strengths of both discrete and continuous approaches.<n>Our approach unifies the theories of discrete and continuous diffusion models, offering a more principled and effective framework for text generation.
  arXiv  Detail & Related papers  (2025-05-28T09:28:52Z)
• CtrlDiff: Boosting Large Diffusion Language Models with Dynamic Block Prediction and Controllable Generation [7.250878248686215]
  Diffusion-based language models have emerged as a compelling alternative due to their powerful parallel generation capabilities and inherent editability.<n>We propose CtrlDiff, a dynamic and controllable semi-autoregressive framework that adaptively determines the size of each generation block based on local semantics.
  arXiv  Detail & Related papers  (2025-05-20T14:52:41Z)
• Generalized Interpolating Discrete Diffusion [65.74168524007484]
  Masked diffusion is a popular choice due to its simplicity and effectiveness.<n>We derive the theoretical backbone of a family of general interpolating discrete diffusion processes.<n>Exploiting GIDD's flexibility, we explore a hybrid approach combining masking and uniform noise.
  arXiv  Detail & Related papers  (2025-03-06T14:30:55Z)
• Interleaved Gibbs Diffusion: Generating Discrete-Continuous Data with Implicit Constraints [30.624303845550575]
  Interleaved Gibbs Diffusion (IGD) is a novel generative modeling framework for discrete-continuous data.<n>IGD generalizes discrete time Gibbs sampling type Markov chain for the case of discrete-continuous generation.<n>It achieves state-of-the-art results without relying on domain-specific inductive biases.
  arXiv  Detail & Related papers  (2025-02-19T05:51:24Z)
• Controlled LLM Decoding via Discrete Auto-regressive Biasing [9.843359827321194]
  Controlled text generation allows for enforcing user-defined constraints on large language model outputs. We propose Discrete Auto-regressive Biasing, a controlled decoding algorithm that leverages gradients while operating entirely in the discrete text domain. Our method significantly improves constraint satisfaction while maintaining comparable or better fluency, all with even lower computational costs.
  arXiv  Detail & Related papers  (2025-02-06T00:14:43Z)
• Simple Guidance Mechanisms for Discrete Diffusion Models [44.377206440698586]
  We develop a new class of diffusion models that leverage uniform noise and that are more guidable because they can continuously edit their outputs.<n>We improve the quality of these models with a novel continuous-time variational lower bound that yields state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-12-13T15:08:30Z)
• Diffusion Predictive Control with Constraints [51.91057765703533]
  Diffusion predictive control with constraints (DPCC)<n>An algorithm for diffusion-based control with explicit state and action constraints that can deviate from those in the training data.<n>We show through simulations of a robot manipulator that DPCC outperforms existing methods in satisfying novel test-time constraints while maintaining performance on the learned control task.
  arXiv  Detail & Related papers  (2024-12-12T15:10:22Z)
• Conditional [MASK] Discrete Diffusion Language Model [14.208510167132983]
  Diffusion-EAGS is a framework that integrates conditional masked language models into diffusion language models.<n>We show that Diffusion-EAGS achieves the best quality-diversity tradeoff, demonstrating its effectiveness in non-autoregressive text generation.
  arXiv  Detail & Related papers  (2024-11-10T11:49:36Z)
• Constrained Diffusion Models via Dual Training [80.03953599062365]
  Diffusion processes are prone to generating samples that reflect biases in a training dataset. We develop constrained diffusion models by imposing diffusion constraints based on desired distributions. We show that our constrained diffusion models generate new data from a mixture data distribution that achieves the optimal trade-off among objective and constraints.
  arXiv  Detail & Related papers  (2024-08-27T14:25:42Z)
• Text Diffusion with Reinforced Conditioning [92.17397504834825]
  This paper thoroughly analyzes text diffusion models and uncovers two significant limitations: degradation of self-conditioning during training and misalignment between training and sampling. Motivated by our findings, we propose a novel Text Diffusion model called TREC, which mitigates the degradation with Reinforced Conditioning and the misalignment by Time-Aware Variance Scaling.
  arXiv  Detail & Related papers  (2024-02-19T09:24:02Z)
• Constrained Synthesis with Projected Diffusion Models [47.56192362295252]
  This paper introduces an approach to generative diffusion processes the ability to satisfy and certify compliance with constraints and physical principles. The proposed method recast the traditional process of generative diffusion as a constrained distribution problem to ensure adherence to constraints.
  arXiv  Detail & Related papers  (2024-02-05T22:18:16Z)
• TESS: Text-to-Text Self-Conditioned Simplex Diffusion [56.881170312435444]
  Text-to-text Self-conditioned Simplex Diffusion employs a new form of self-conditioning, and applies the diffusion process on the logit simplex space rather than the learned embedding space. We demonstrate that TESS outperforms state-of-the-art non-autoregressive models, requires fewer diffusion steps with minimal drop in performance, and is competitive with pretrained autoregressive sequence-to-sequence models.
  arXiv  Detail & Related papers  (2023-05-15T06:33:45Z)
• A Cheaper and Better Diffusion Language Model with Soft-Masked Noise [62.719656543880596]
  Masked-Diffuse LM is a novel diffusion model for language modeling, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. We demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.
  arXiv  Detail & Related papers  (2023-04-10T17:58:42Z)
• Conditional Hybrid GAN for Sequence Generation [56.67961004064029]
  We propose a novel conditional hybrid GAN (C-Hybrid-GAN) to solve this issue. We exploit the Gumbel-Softmax technique to approximate the distribution of discrete-valued sequences. We demonstrate that the proposed C-Hybrid-GAN outperforms the existing methods in context-conditioned discrete-valued sequence generation.
  arXiv  Detail & Related papers  (2020-09-18T03:52:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.