A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Models

• URL: http://arxiv.org/abs/2609.06324v2
• Date: Fri, 11 Sep 2026 18:39:03 GMT
• Title: A Ticket from Marginals to Joints: Coupled-Noise Distillation for One-Step Block Generation in Diffusion Language Models
• Abstract summary: We ask whether a discrete masked model can commit an entire block in one pass when its mask embeddings are perturbed by a sampled Gaussian noise field.<n>We propose CONDOR, which trains such a model from scratch without a target-side encoder or an autoregressive teacher.<n>On a controlled TinyStories setting, this recipe yields coherent one-pass continuations that vary with the noise.
• Score: 0.6916773850242582
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Diffusion language models (dLLMs) predict all tokens of a block in parallel, but a single forward pass samples each position from its own marginal distribution, so the tokens need not form a coherent block. We ask whether a discrete masked model can commit an entire block in one pass when its mask embeddings are perturbed by a sampled Gaussian noise field: the same noise should give the same coherent continuation, and different noise should give different ones. We propose CONDOR (Coupled-Noise Distillation for One-Step Readout), which trains such a model from scratch without a target-side encoder or an autoregressive teacher. Training combines two signals. On real text, the model predicts masked tokens under several noise samples and is supervised only through the sample that fits the ground truth best, so different noise can specialize to different continuations. For the remaining samples, the model refines its own one-pass prediction over several decoding steps under the same fixed noise and then distills that refined block back into a single pass. On a controlled TinyStories setting, this recipe yields coherent one-pass continuations that vary with the noise, both for a single block and, with a block-causal variant, when blocks are generated one after another.

Related papers

• Latent-Kernel Discrete Flow Maps for Few-Step Generation [4.159429302112398]
  We build a flow-map kernel that is a mixture of M factorized components tied by a single latent.<n>We show that a single step places mass on correlated completions with the same sampling time complexity as a factorized model.<n>Experiments for unconditional text generation on the One-Billion-Word (LM1B) and WikiText-103 benchmarks show that our LKF model learns strongly heterogeneous components.
  arXiv  Detail & Related papers  (2026-07-29T23:46:31Z)
• DSL-LLaDA: Scaling Continuous Denoising to 8B Masked Diffusion LMs [29.561307941602482]
  We show that a pretrained masked DLM can be lightly adapted to support continuous embedding-space denoising.<n>The adapted model supports continuous inference that evolves all positions jointly in embedding space.<n> DSL-LLaDA-SDE achieves the best ROUGE-1 on all four benchmarks.
  arXiv  Detail & Related papers  (2026-05-31T05:27:01Z)
• Multi-Token Residual Prediction [21.234596379356724]
  Diffusion Language Models generate text by iteratively denoising masked token sequences.<n>We introduce Multi-token Residual Prediction (MRP), a lightweight module that enables dependency-aware multi-token denoising.<n>MRP predicts the residual between steps from the backbone's hidden states, effectively denoising more tokens per backbone forward at a fraction of the cost.
  arXiv  Detail & Related papers  (2026-05-12T11:40:39Z)
• Rejection Mixing: Fast Semantic Propagation of Mask Tokens for Efficient DLLM Inference [58.189320101488725]
  DLLMs promise fast non-autoregressive inference but suffer a severe quality-speed trade-off in parallel decoding.<n>We address this by integrating continuous representations into the discrete decoding process, as they preserve rich inter-position dependency.<n>We propose ReMix, a framework that introduces a novel Continuous Mixing State as an intermediate between the initial masked state and the final decoded token state.
  arXiv  Detail & Related papers  (2026-02-26T11:08:11Z)
• Combating Noisy Labels through Fostering Self- and Neighbor-Consistency [120.4394402099635]
  Label noise is pervasive in various real-world scenarios, posing challenges in supervised deep learning.<n>We propose a noise-robust method named Jo-SNC (textbfJoint sample selection and model regularization based on textbfSelf- and textbfNeighbor-textbfConsistency)<n>We design a self-adaptive, data-driven thresholding scheme to adjust per-class selection thresholds.
  arXiv  Detail & Related papers  (2026-01-19T07:55:29Z)
• Stratified Hazard Sampling: Minimal-Variance Event Scheduling for CTMC/DTMC Discrete Diffusion and Flow Models [0.0]
  Stratified Hazard Sampling Sampling (SHS) models per-token edits as events driven by cumulative hazard (CTMC) or cumulative jump mass (DTMC)<n>SHS models per-token edits as events driven by cumulative hazard (CTMC) or cumulative jump mass (DTMC) and places events by stratifying this cumulative quantity.<n>We also introduce a phase-allocation variant for blacklist-style lexical constraints that prioritizes early edits at high-risk positions to mitigate late-masking artifacts.
  arXiv  Detail & Related papers  (2026-01-06T08:19:02Z)
• DiSTAR: Diffusion over a Scalable Token Autoregressive Representation for Speech Generation [30.150846119894577]
  DISTAR is a zero-shot text-to-speech framework that operates entirely in a discrete residual vector quantization code space.<n>Experiments and ablations demonstrate that DISTAR surpasses state-of-the-art zero-shot TTS systems in robustness, naturalness, and speaker/style consistency.
  arXiv  Detail & Related papers  (2025-10-14T07:03:29Z)
• Enabling Approximate Joint Sampling in Diffusion LMs [16.729589221035074]
  Masked diffusion language models generate text by unmasking tokens out of order and potentially in parallel.<n>In this paper we devise a way to em approximately sample multiple tokens from the joint distribution in a single full-model forward pass.
  arXiv  Detail & Related papers  (2025-09-25T21:23:26Z)
• Blockwise SFT for Diffusion Language Models: Reconciling Bidirectional Attention and Autoregressive Decoding [60.06816407728172]
  Discrete diffusion language models have shown strong potential for text generation.<n>Standard supervised fine-tuning misaligns with semi-autoregressive inference.<n>We propose Blockwise SFT, which partitions responses into fixed-size blocks.
  arXiv  Detail & Related papers  (2025-08-27T02:49:33Z)
• Autoregressive Speech Synthesis without Vector Quantization [135.4776759536272]
  We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS)<n>MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition.<n>MELLE mitigates robustness issues by avoiding the inherent flaws of sampling vector-quantized codes.
  arXiv  Detail & Related papers  (2024-07-11T14:36:53Z)
• Diffusion Forcing: Next-token Prediction Meets Full-Sequence Diffusion [61.03681839276652]
  Diffusion Forcing is a new training paradigm where a diffusion model is trained to denoise a set of tokens with independent per-token noise levels.<n>We apply Diffusion Forcing to sequence generative modeling by training a causal next-token prediction model to generate one or several future tokens.
  arXiv  Detail & Related papers  (2024-07-01T15:43:25Z)
• ROG$_{PL}$: Robust Open-Set Graph Learning via Region-Based Prototype Learning [52.60434474638983]
  We propose a unified framework named ROG$_PL$ to achieve robust open-set learning on complex noisy graph data. The framework consists of two modules, i.e., denoising via label propagation and open-set prototype learning via regions. To the best of our knowledge, the proposed ROG$_PL$ is the first robust open-set node classification method for graph data with complex noise.
  arXiv  Detail & Related papers  (2024-02-28T17:25:06Z)
• BASS: Block-wise Adaptation for Speech Summarization [47.518484305407185]
  We develop a method that allows one to train summarization models on very long sequences in an incremental manner. Speech summarization is realized as a streaming process, where hypothesis summaries are updated every block. Experiments on the How2 dataset demonstrate that the proposed block-wise training method improves by 3 points absolute on ROUGE-L over a truncated input baseline.
  arXiv  Detail & Related papers  (2023-07-17T03:31:36Z)
• Transferring Annotator- and Instance-dependent Transition Matrix for Learning from Crowds [88.06545572893455]
  In real-world crowd-sourcing scenarios, noise transition matrices are both annotator- and instance-dependent. We first model the mixture of noise patterns by all annotators, and then transfer this modeling to individual annotators. Experiments confirm the superiority of the proposed approach on synthetic and real-world crowd-sourcing data.
  arXiv  Detail & Related papers  (2023-06-05T13:43:29Z)
• Learning Noise-Aware Encoder-Decoder from Noisy Labels by Alternating Back-Propagation for Saliency Detection [54.98042023365694]
  We propose a noise-aware encoder-decoder framework to disentangle a clean saliency predictor from noisy training examples. The proposed model consists of two sub-models parameterized by neural networks.
  arXiv  Detail & Related papers  (2020-07-23T18:47:36Z)
• Confidence Scores Make Instance-dependent Label-noise Learning Possible [129.84497190791103]
  In learning with noisy labels, for every instance, its label can randomly walk to other classes following a transition distribution which is named a noise model. We introduce confidence-scored instance-dependent noise (CSIDN), where each instance-label pair is equipped with a confidence score. We find with the help of confidence scores, the transition distribution of each instance can be approximately estimated.
  arXiv  Detail & Related papers  (2020-01-11T16:15:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.