Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design

• URL: http://arxiv.org/abs/2606.15327v1
• Date: Sat, 13 Jun 2026 14:41:55 GMT
• Title: Semantic DLM+: Improving Diffusion Language Models through Bias-variance Trade-off in Transition Kernel Design
• Abstract summary: Diffusion Language Models (DLMs) have demonstrated strong scaling capacity as alternatives to autoregressive language models.<n>We study this sensitivity through a principled analysis of generalization error and identify three critical factors.<n>We propose SemDLM+, which adds a global transition and a semantic-frequency penalty during sampling.
• Score: 59.05127237532803
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Diffusion Language Models (DLMs) have demonstrated strong scaling capacity as alternatives to autoregressive language models. However, their performance is highly sensitive to the choice of transition kernels, and poorly designed kernels can lead to issues like training instability, slow convergence, and biased sampling. In this paper, we study this sensitivity through a principled analysis of generalization error and identify three critical factors: asymptotic bias (difficulty in approximating the posterior distribution), exposure bias (error propagation during sampling), and optimization variance induced by kernel dispersion. We further compare different transition kernels: masking diffusion yields sparse and easier posterior-approximation targets, while uniform diffusion provides stronger sampling-side repair but induces harder approximation. Motivated by this trade-off, we revisit a previously overlooked variant, semantic DLM (SemDLM), where the transition kernel corrupts tokens to neighborhoods that are semantically similar. Our theory suggests that SemDLM can serve as a plausible middle ground by reducing the posterior approximation difficulty of uniform diffusion while retaining repair ability. However, we find that SemDLM suffers from a semantic basin problem, where sampling repeatedly stays within a semantic region and produces low-diversity text. To address this, we propose SemDLM+, which adds a global transition and a semantic-frequency penalty during sampling. Experiments on LM1B and OpenWebText show that SemDLM+ improves training dynamics and achieves competitive language modeling and generation quality with satisfactory diversity.

Related papers

• MLLM-Guided Semantic Correction for Text-to-Video Generation [66.7723669725808]
  We introduce a training-free, interpretable mid-generation correction framework that integrates multimodal large language model feedback.<n>We propose two key modules: a Semantic Assessment Supervisor that generates intermediate preview frames for semantic evaluations and deviation diagnostics, and a Semantic Modification Assistant that corrects semantic drift during inference.<n>Our method improves semantic alignment, visual fidelity, and temporal consistency without modifying model parameters.
  arXiv  Detail & Related papers  (2026-08-17T12:54:10Z)
• Continuous Diffusion Scales Competitively with Discrete Diffusion for Language [48.290726603587444]
  We construct RePlaid as a likelihood-based continuous diffusion language model (DLM)<n>We show that RePlaid exhibits a compute gap of only $20times$ compared to autoregressive models.<n>We also offer theoretical insights to understand the advantage of likelihood-based training.
  arXiv  Detail & Related papers  (2026-05-18T15:15:24Z)
• Stochastic Transition-Map Distillation for Fast Probabilistic Inference [14.377745343756485]
  Diffusion models achieve strong generation quality, diversity, and distribution coverage, but their performance often comes with expensive inference.<n>In this work, we propose Transition-Map Distillation (STMD), a teacher-free framework for accelerating diffusion model inference.
  arXiv  Detail & Related papers  (2026-05-08T12:31:44Z)
• Consistent Diffusion Language Models [22.253153649144476]
  Diffusion language models (DLMs) promise sub-time, parallel generation, yet practical gains remain elusive.<n>In continuous domains, consistency training along the probability-flow ODE is a popular recipe to accelerate diffusion.<n>We introduce Multi-Path Discrete Consistency (DCMP), a new principle that trains a denoiser to be path-in expectation across these bridges.<n>A single CDLM objective unifies masked diffusion, continuous consistency models, and progressive/discrete distillation as analytic limits or empirical approximations of one common view.
  arXiv  Detail & Related papers  (2026-04-30T19:31:02Z)
• IDLM: Inverse-distilled Diffusion Language Models [70.5793829229702]
  We extend Inverse Distillation, a technique originally developed to accelerate continuous diffusion models, to the discrete setting.<n>From a theoretical perspective, the inverse distillation objective lacks uniqueness guarantees, which may lead to suboptimal solutions.<n>We show that our method, Inverse-distilled Diffusion Language Models (IDLM), reduces the number of inference steps by 4x-64x.
  arXiv  Detail & Related papers  (2026-02-22T06:47:04Z)
• Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models [6.4453302264198165]
  We exploit the flexibility of the diffusion framework and study model scheduling, where a smaller MDLM replaces the full model at a subset of denoising steps.<n>We show that early and late denoising steps are substantially more robust to such replacement than middle steps, enabling up to a 17% reduction in FLOPs.<n>Our results suggest that simple, architecture-agnostic scheduling rules can significantly accelerate MDLM sampling while largely preserving generation quality.
  arXiv  Detail & Related papers  (2026-02-04T13:04:58Z)
• An Effective Deployment of Diffusion LM for Data Augmentation in Low-Resource Sentiment Classification [2.0930389307057427]
  Sentiment classification (SC) often suffers from low-resource challenges such as domain-specific contexts, imbalanced label distributions, and few-shot scenarios. We propose Diffusion LM to capture in-domain knowledge and generate pseudo samples by reconstructing strong label-related tokens.
  arXiv  Detail & Related papers  (2024-09-05T02:51:28Z)
• Denoising Diffusion Variational Inference: Diffusion Models as Expressive Variational Posteriors [11.01598521921903]
  DDVI is a black-box variational inference algorithm for latent variable models.<n>We introduce an expressive class of diffusion-based variational posteriors.<n>We train these posteriors with a novel regularized evidence lower bound.
  arXiv  Detail & Related papers  (2024-01-05T10:27:44Z)
• On Error Propagation of Diffusion Models [77.91480554418048]
  We develop a theoretical framework to mathematically formulate error propagation in the architecture of DMs. We apply the cumulative error as a regularization term to reduce error propagation. Our proposed regularization reduces error propagation, significantly improves vanilla DMs, and outperforms previous baselines.
  arXiv  Detail & Related papers  (2023-08-09T15:31:17Z)
• A Cheaper and Better Diffusion Language Model with Soft-Masked Noise [62.719656543880596]
  Masked-Diffuse LM is a novel diffusion model for language modeling, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. We demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.
  arXiv  Detail & Related papers  (2023-04-10T17:58:42Z)
• DensePure: Understanding Diffusion Models towards Adversarial Robustness [110.84015494617528]
  We analyze the properties of diffusion models and establish the conditions under which they can enhance certified robustness. We propose a new method DensePure, designed to improve the certified robustness of a pretrained model (i.e. a classifier) We show that this robust region is a union of multiple convex sets, and is potentially much larger than the robust regions identified in previous works.
  arXiv  Detail & Related papers  (2022-11-01T08:18:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.