DLM-SWAI: Steering Diffusion Language Models Before They Unmask

• URL: http://arxiv.org/abs/2605.29626v1
• Date: Thu, 28 May 2026 09:00:14 GMT
• Title: DLM-SWAI: Steering Diffusion Language Models Before They Unmask
• Abstract summary: DLM-SWAI is a training-free steering method that biases the token distribution at each denoising step using pre-computed token-level style scores.<n> Experiments on style and safety control tasks show that DLM-SWAI effectively steers diffusion language models while preserving generation quality and requiring minimal computational overhead.
• Score: 5.276802062716119
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Steering language model generation toward desired textual properties is essential for practical deployment, and inference-time methods are particularly appealing because they enable controllable generation without retraining. Recent work has also highlighted diffusion language models as an emerging generation paradigm with distinct decoding properties. However, most existing steering approaches either rely on auxiliary models or are designed for autoregressive next-token decoding, making them difficult to apply to diffusion language models DLMs, which generate text through iterative denoising of partially masked sequences. Therefore, we propose DLM-SWAI, a simple training-free steering method that biases the token distribution at each denoising step using pre-computed token-level style scores. Experiments on style and safety control tasks show that DLM-SWAI effectively steers diffusion language models while preserving generation quality and requiring minimal computational overhead. Ablations further reveal a controllable trade-off between steering strength and fluency, and our analysis links class-wise steerability to the strength of token-level attribute cues.

Related papers

• AURORA-LM: Autoencoding Unified Representation for Continuous-Latent Diffusion Language Modeling [60.04959047453115]
  AURORA-LM is a continuous-latent language model that separates the construction of a decodable text representation from the modeling of its distribution.<n>AURORA-LM achieves the strongest performance among evaluated continuous and diffusion-based language models on OpenWebText free generation and XSum summarization.
  arXiv  Detail & Related papers  (2026-08-03T17:59:50Z)
• How to Train Your Latent Diffusion Language Model Jointly With the Latent Space [76.9057986588963]
  Latent diffusion models offer an attractive alternative to discrete diffusion for non-autoregressive text generation.<n>We present the Latent Diffusion Language Model (LDLM), in which the latent encoder, diffusion model, and decoder are trained jointly.<n>On OpenWebText and LM1B, LDLM achieves better generation performance than existing discrete and continuous diffusion language models.
  arXiv  Detail & Related papers  (2026-05-08T16:05:19Z)
• Diffutron: A Masked Diffusion Language Model for Turkish Language [0.0]
  We introduce $textitDiffutron$, a masked diffusion language model specifically designed for Turkish.<n>We employ a progressive instruction-tuning strategy, sequentially adapting the model on general and task-specific instruction sets.<n>These findings validate the effectiveness of masked diffusion modeling combined with multi-stage tuning for non-autoregressive text generation in Turkish.
  arXiv  Detail & Related papers  (2026-03-20T19:52:37Z)
• Activation Steering for Masked Diffusion Language Models [1.0980666029958932]
  Masked diffusion language models generate text through an iterative denoising process.<n>We present an activation-steering framework for MDLMs that computes layer-wise steering vectors from a single forward pass.<n>Experiments on LLaDA-8B-Instruct demonstrate reliable modulation of high-level attributes.
  arXiv  Detail & Related papers  (2025-12-30T11:10:52Z)
• WeMMU: Enhanced Bridging of Vision-Language Models and Diffusion Models via Noisy Query Tokens [69.97021957331326]
  We propose Noisy Query Tokens, which learn a distributed representation space between the VLM and Diffusion Model via end-to-end optimization.<n>We also introduce a VAE branch with linear projection to recover fine-grained image details.
  arXiv  Detail & Related papers  (2025-12-02T09:02:20Z)
• Improving Text Style Transfer using Masked Diffusion Language Models with Inference-time Scaling [37.795834398730555]
  Masked diffusion language models (MDMs) have recently gained traction as a viable generative framework for natural language.<n>We propose a verifier-based inference-time scaling method that aids in finding a better candidate generation during the denoising process of the MDM.<n>Our experiments demonstrate the application of MDMs for standard text-style transfer tasks and establish MDMs as a better alternative to autoregressive language models.
  arXiv  Detail & Related papers  (2025-08-14T18:01:22Z)
• TreeDiff: AST-Guided Code Generation with Diffusion LLMs [27.111814602726227]
  We propose a syntax-aware diffusion framework that incorporates structural priors from Abstract Syntax Trees (ASTs) into the denoising process.<n>Results demonstrate that syntax-aware corruption significantly improves syntactic correctness, reconstruction accuracy, and generalization to unseen code patterns.
  arXiv  Detail & Related papers  (2025-08-02T19:46:09Z)
• A Cheaper and Better Diffusion Language Model with Soft-Masked Noise [62.719656543880596]
  Masked-Diffuse LM is a novel diffusion model for language modeling, inspired by linguistic features in languages. Specifically, we design a linguistic-informed forward process which adds corruptions to the text through strategically soft-masking to better noise the textual data. We demonstrate that our Masked-Diffuse LM can achieve better generation quality than the state-of-the-art diffusion models with better efficiency.
  arXiv  Detail & Related papers  (2023-04-10T17:58:42Z)
• GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator [114.8954615026781]
  We propose a GAN-style model for encoder-decoder pre-training by introducing an auxiliary discriminator. GanLM is trained with two pre-training objectives: replaced token detection and replaced token denoising. Experiments in language generation benchmarks show that GanLM with the powerful language understanding capability outperforms various strong pre-trained language models.
  arXiv  Detail & Related papers  (2022-12-20T12:51:11Z)
• Unsupervised Paraphrasing with Pretrained Language Models [85.03373221588707]
  We propose a training pipeline that enables pre-trained language models to generate high-quality paraphrases in an unsupervised setting. Our recipe consists of task-adaptation, self-supervision, and a novel decoding algorithm named Dynamic Blocking. We show with automatic and human evaluations that our approach achieves state-of-the-art performance on both the Quora Question Pair and the ParaNMT datasets.
  arXiv  Detail & Related papers  (2020-10-24T11:55:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.