A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Models

• URL: http://arxiv.org/abs/2604.09921v1
• Date: Fri, 10 Apr 2026 21:36:01 GMT
• Title: A Tale of Two Temperatures: Simple, Efficient, and Diverse Sampling from Diffusion Language Models
• Abstract summary: We show that diversity can be increased by using softened, tempered versions of familiar confidence-based remaskings.<n>Our findings demonstrate that simple, efficient, and diverse sampling from dLLMs is possible.
• Score: 25.248923429171498
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Much work has been done on designing fast and accurate sampling for diffusion language models (dLLMs). However, these efforts have largely focused on the tradeoff between speed and quality of individual samples; how to additionally ensure diversity across samples remains less well understood. In this work, we show that diversity can be increased by using softened, tempered versions of familiar confidence-based remasking heuristics, retaining their computational benefits and offering simple implementations. We motivate this approach by introducing an idealized formal model of fork tokens and studying the impact of remasking on the expected entropy at the forks. Empirically, the proposed tempered heuristics close the exploration gap (pass@k) between existing confidence-based and autoregressive sampling, hence outperforming both when controlling for cost (pass@NFE). We further study how the increase in diversity translates to downstream post-training and test-time compute scaling. Overall, our findings demonstrate that simple, efficient, and diverse sampling from dLLMs is possible.

Related papers

• Free Lunch for Pass@$k$? Low Cost Diverse Sampling for Diffusion Language Models [17.37935640125399]
  We propose a training free, low cost intervention to enhance generative diversity in Diffusion Language Models.<n>Our approach modifies intermediate samples in a batch sequentially, where each sample is repelled from the feature space of previous samples.<n>Unlike prior methods that require retraining or beam search, our strategy incurs negligible computational overhead.
  arXiv  Detail & Related papers  (2026-03-05T07:35:07Z)
• Sharp Convergence Rates for Masked Diffusion Models [53.117058231393834]
  We develop a total-variation based analysis for the Euler method that overcomes limitations.<n>Our results relax assumptions on score estimation, improve parameter dependencies, and establish convergence guarantees.<n>Overall, our analysis introduces a direct TV-based error decomposition along the CTMC trajectory and a decoupling-based path-wise analysis for FHS.
  arXiv  Detail & Related papers  (2026-02-26T00:47:51Z)
• KLASS: KL-Guided Fast Inference in Masked Diffusion Models [31.082562509845985]
  Masked diffusion models have demonstrated competitive results on various tasks including language generation.<n>We introduce KL-Adaptive Stability Sampling' (KLASS), a fast yet effective sampling method that exploits token-level KL divergence to identify stable, high-confidence predictions.<n>On reasoning benchmarks, KLASS achieves up to $2.78times$ wall-clock speedups while improving performance over standard greedy decoding.
  arXiv  Detail & Related papers  (2025-11-07T19:05:36Z)
• Temporal Score Rescaling for Temperature Sampling in Diffusion and Flow Models [30.36841165328262]
  We present a mechanism to steer the sampling diversity of flow matching models.<n>We show that rescaling these allows one to effectively control a local' sampling temperature.
  arXiv  Detail & Related papers  (2025-10-01T17:59:51Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• Mitigating Shortcut Learning with Diffusion Counterfactuals and Diverse Ensembles [104.60508550106618]
  We propose DiffDiv, an ensemble diversification framework exploiting Diffusion Probabilistic Models (DPMs)<n>We show that DPMs can generate images with novel feature combinations, even when trained on samples displaying correlated input features.<n>We show that DPM-guided diversification is sufficient to remove dependence on shortcut cues, without a need for additional supervised signals.
  arXiv  Detail & Related papers  (2023-11-23T15:47:33Z)
• Leveraging Diffusion Disentangled Representations to Mitigate Shortcuts in Underspecified Visual Tasks [92.32670915472099]
  We propose an ensemble diversification framework exploiting the generation of synthetic counterfactuals using Diffusion Probabilistic Models (DPMs) We show that diffusion-guided diversification can lead models to avert attention from shortcut cues, achieving ensemble diversity performance comparable to previous methods requiring additional data collection.
  arXiv  Detail & Related papers  (2023-10-03T17:37:52Z)
• Model ensemble instead of prompt fusion: a sample-specific knowledge transfer method for few-shot prompt tuning [85.55727213502402]
  We focus on improving the few-shot performance of prompt tuning by transferring knowledge from soft prompts of source tasks. We propose Sample-specific Ensemble of Source Models (SESoM) SESoM learns to adjust the contribution of each source model for each target sample separately when ensembling source model outputs.
  arXiv  Detail & Related papers  (2022-10-23T01:33:16Z)
• Multi-Scale Positive Sample Refinement for Few-Shot Object Detection [61.60255654558682]
  Few-shot object detection (FSOD) helps detectors adapt to unseen classes with few training instances. We propose a Multi-scale Positive Sample Refinement (MPSR) approach to enrich object scales in FSOD. MPSR generates multi-scale positive samples as object pyramids and refines the prediction at various scales.
  arXiv  Detail & Related papers  (2020-07-18T09:48:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.