Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models

• URL: http://arxiv.org/abs/2605.10971v1
• Date: Fri, 08 May 2026 18:52:17 GMT
• Title: Steering Without Breaking: Mechanistically Informed Interventions for Discrete Diffusion Language Models
• Abstract summary: Discrete diffusion language models (DLMs) generate text by iteratively denoising all positions in parallel.<n>We show this uniform schedule degrades quality, and the damage compounds when multiple attributes are steered jointly.<n>We propose a novel adaptive scheduler that concentrates interventions on the steps where an attribute is actively forming and leaves the rest of generation untouched.
• Score: 10.45663857615872
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discrete diffusion language models (DLMs) generate text by iteratively denoising all positions in parallel, offering an alternative to autoregressive models. Controlled generation methods for DLMs, imported from autoregressive models, apply uniform intervention at every denoising steps. We show this uniform schedule degrades quality, and the damage compounds when multiple attributes are steered jointly. To diagnose the failure, we train sparse autoencoders on four DLMs (124M-8B parameters) and find that different attributes commit on distinct schedules, varying in timing, sharpness, and magnitude. For instance, topic commits within the first 2\% of denoising, whereas sentiment emerges gradually over 20\% of the process. Consequently, uniform intervention wastes steering capacity on steps where the target attribute has already solidified or has yet to emerge. We propose a novel adaptive scheduler that concentrates interventions on the steps where an attribute is actively forming and leaves the rest of generation untouched. The cost-control trade-off admits a closed-form characterization: the advantage of adaptive over uniform scheduling is governed by a single dispersion statistic of the commitment distribution. Across four DLMs and seven steering tasks, our method achieves precise control without the degradation typical of uniform interventions. Especially on challenging simultaneous three-attribute control, it reaches up to 93\% steering strength, beating the strongest baseline by up to 15\% points while preserving generation quality.

Related papers

• Uncertainty DMD: Restoring Diversity in Few-Step Autoregressive Video Distillation [52.34982929108336]
  Few-step distillation improves the efficiency of autoregressive (AR) video generation, but often causes diversity collapse.<n>We analyze this degradation in Distribution Matching Distillation (DMD)-distilled AR video generators and find that, in the autoregressive setting, it takes the form of a structured uncertainty collapse.
  arXiv  Detail & Related papers  (2026-09-10T08:59:47Z)
• Temporal Difference Learning for Diffusion Models [19.82560792699663]
  We introduce a temporal difference (TD) objective that penalizes inconsistency of the model's multi-step progress along the denoising path.<n>We propose a principled sample-based reweighting method that stabilizes training.
  arXiv  Detail & Related papers  (2026-06-13T01:34:04Z)
• Adaptive Steering and Remasking for Safe Generation in Diffusion Language Models [6.107855416893071]
  Diffusion Language Models (DLMs) provide a promising alternative to autoregressive language models.<n>In this paper, we propose an inference-time defense framework based on the step-wise intervention during the denoising process.<n>Our approaches reduce jailbreak success rates to 0.64% while preserving generation quality close to the original model performance.
  arXiv  Detail & Related papers  (2026-05-13T05:56:45Z)
• Inference-Time Attribute Distribution Alignment for Unconditional Diffusion [54.28942858023809]
  We formalize the inference-time attribute distributional alignment problem for pretrained unconditional diffusion models.<n>To address this, we cast inference-time attribute distributional alignment as an optimal control problem over the reverse diffusion process.<n>We solve for the perturbations using an optimal-control-based algorithm to optimize a differentiable distribution-matching objective.
  arXiv  Detail & Related papers  (2026-05-08T09:02:47Z)
• Interpolating Discrete Diffusion Models with Controllable Resampling [49.771004594828405]
  Masked diffusion models suffer from irreversible errors due to early unmasking.<n>Uniform diffusion models, despite enabling self-correction, often yield low-quality samples.<n>We introduce Interpolating Discrete Diffusion Model, that improves diffusion by reducing dependence on intermediate latent states.
  arXiv  Detail & Related papers  (2026-04-19T07:55:57Z)
• OmniForcing: Unleashing Real-time Joint Audio-Visual Generation [51.031726911410594]
  Recent joint audio-visual diffusion models achieve remarkable generation quality but suffer from high latency.<n>We propose OmniForcing, the first framework to distill an offline, dual-stream bidirectional diffusion model into a high-fidelity streaming autore generator.
  arXiv  Detail & Related papers  (2026-03-12T08:17:36Z)
• Not All Denoising Steps Are Equal: Model Scheduling for Faster Masked Diffusion Language Models [6.4453302264198165]
  We exploit the flexibility of the diffusion framework and study model scheduling, where a smaller MDLM replaces the full model at a subset of denoising steps.<n>We show that early and late denoising steps are substantially more robust to such replacement than middle steps, enabling up to a 17% reduction in FLOPs.<n>Our results suggest that simple, architecture-agnostic scheduling rules can significantly accelerate MDLM sampling while largely preserving generation quality.
  arXiv  Detail & Related papers  (2026-02-04T13:04:58Z)
• Inference-time Alignment via Sparse Junction Steering [25.464612964225484]
  Token-level steering has emerged as a pivotal approach for inference-time alignment.<n>Existing methods rely on dense intervention at every decoding step.<n>We show that dense intervention is unnecessary and propose sparse junction steering.
  arXiv  Detail & Related papers  (2026-01-30T08:40:47Z)
• Continuously Augmented Discrete Diffusion model for Categorical Generative Modeling [87.34677262370924]
  Standard discrete diffusion models treat all unobserved states identically by mapping them to an absorbing [MASK] token.<n>This creates an 'information void' where semantic information that could be inferred from unmasked tokens is lost between denoising steps.<n>We introduce Continuously Augmented Discrete Diffusion, a framework that augments the discrete state space with a paired diffusion in a continuous latent space.
  arXiv  Detail & Related papers  (2025-10-01T18:00:56Z)
• Hybrid Autoregressive-Diffusion Model for Real-Time Sign Language Production [0.0]
  We develop a hybrid approach that combines autoregressive and diffusion models for Sign Language Production (SLP)<n>To capture fine-grained body movements, we design a Multi-Scale Pose Representation module that separately extracts detailed features from distinct articulators.<n>We introduce a Confidence-Aware Causal Attention mechanism that utilizes joint-level confidence scores to dynamically guide the pose generation process.
  arXiv  Detail & Related papers  (2025-07-12T01:34:50Z)
• Continual-MAE: Adaptive Distribution Masked Autoencoders for Continual Test-Time Adaptation [49.827306773992376]
  Continual Test-Time Adaptation (CTTA) is proposed to migrate a source pre-trained model to continually changing target distributions. Our proposed method attains state-of-the-art performance in both classification and segmentation CTTA tasks.
  arXiv  Detail & Related papers  (2023-12-19T15:34:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.