Free Lunch for Pass@$k$? Low Cost Diverse Sampling for Diffusion Language Models

• URL: http://arxiv.org/abs/2603.04893v1
• Date: Thu, 05 Mar 2026 07:35:07 GMT
• Title: Free Lunch for Pass@$k$? Low Cost Diverse Sampling for Diffusion Language Models
• Authors: Sean Lamont, Christian Walder, Paul Montague, Amir Dezfouli, Michael Norrish,
• Abstract summary: We propose a training free, low cost intervention to enhance generative diversity in Diffusion Language Models.<n>Our approach modifies intermediate samples in a batch sequentially, where each sample is repelled from the feature space of previous samples.<n>Unlike prior methods that require retraining or beam search, our strategy incurs negligible computational overhead.
• Score: 17.37935640125399
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Diverse outputs in text generation are necessary for effective exploration in complex reasoning tasks, such as code generation and mathematical problem solving. Such Pass@$k$ problems benefit from distinct candidates covering the solution space. However, traditional sampling approaches often waste computational resources on repetitive failure modes. While Diffusion Language Models have emerged as a competitive alternative to the prevailing Autoregressive paradigm, they remain susceptible to this redundancy, with independent samples frequently collapsing into similar modes. To address this, we propose a training free, low cost intervention to enhance generative diversity in Diffusion Language Models. Our approach modifies intermediate samples in a batch sequentially, where each sample is repelled from the feature space of previous samples, actively penalising redundancy. Unlike prior methods that require retraining or beam search, our strategy incurs negligible computational overhead, while ensuring that each sample contributes a unique perspective to the batch. We evaluate our method on the HumanEval and GSM8K benchmarks using the LLaDA-8B-Instruct model. Our results demonstrate significantly improved diversity and Pass@$k$ performance across various temperature settings. As a simple modification to the sampling process, our method offers an immediate, low-cost improvement for current and future Diffusion Language Models in tasks that benefit from diverse solution search. We make our code available at https://github.com/sean-lamont/odd.

Related papers

• Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching [66.39914384073145]
  We propose a self-consistency framework that turns cheap diffusion-sampled reasoning into a reusable pool of step-level candidates.<n>We find that step-level recombination is most beneficial on harder problems.<n>Our training-free framework improves average accuracy by up to 2 across six math and coding tasks.
  arXiv  Detail & Related papers  (2026-02-26T11:08:39Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• DiverseFlow: Sample-Efficient Diverse Mode Coverage in Flows [0.6138671548064355]
  DiverseFlow is a training-free approach to improve the diversity of flow models.<n>We demonstrate the efficacy of our method for tasks where sample-efficient diversity is desirable.
  arXiv  Detail & Related papers  (2025-04-10T16:09:50Z)
• Single-Step Consistent Diffusion Samplers [8.758218443992467]
  Existing sampling algorithms typically require many iterative steps to produce high-quality samples.<n>We introduce consistent diffusion samplers, a new class of samplers designed to generate high-fidelity samples in a single step.<n>We show that our approach yields high-fidelity samples using less than 1% of the network evaluations required by traditional diffusion samplers.
  arXiv  Detail & Related papers  (2025-02-11T14:25:52Z)
• Language Rectified Flow: Advancing Diffusion Language Generation with Probabilistic Flows [53.31856123113228]
  This paper proposes Language Rectified Flow (ours) Our method is based on the reformulation of the standard probabilistic flow models. Experiments and ablation studies demonstrate that our method can be general, effective, and beneficial for many NLP tasks.
  arXiv  Detail & Related papers  (2024-03-25T17:58:22Z)
• Improved off-policy training of diffusion samplers [93.66433483772055]
  We study the problem of training diffusion models to sample from a distribution with an unnormalized density or energy function.<n>We benchmark several diffusion-structured inference methods, including simulation-based variational approaches and off-policy methods.<n>Our results shed light on the relative advantages of existing algorithms while bringing into question some claims from past work.
  arXiv  Detail & Related papers  (2024-02-07T18:51:49Z)
• Diffusion-Generative Multi-Fidelity Learning for Physical Simulation [24.723536390322582]
  We develop a diffusion-generative multi-fidelity learning method based on differential equations (SDE), where the generation is a continuous denoising process. By conditioning on additional inputs (temporal or spacial variables), our model can efficiently learn and predict multi-dimensional solution arrays.
  arXiv  Detail & Related papers  (2023-11-09T18:59:05Z)
• Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Models [54.1843419649895]
  We propose a solution based on denoising diffusion probabilistic models (DDPMs) Our motivation for choosing diffusion models over other generative models comes from the flexible internal structure of diffusion models. Our method can unite multiple diffusion models trained on multiple sub-tasks and conquer the combined task.
  arXiv  Detail & Related papers  (2022-12-01T18:59:55Z)
• Arithmetic Sampling: Parallel Diverse Decoding for Large Language Models [65.52639709094963]
  Methods such as beam search and Gumbel top-k sampling can guarantee a different output for each element of the beam, but are not easy to parallelize. We present a framework for sampling according to an arithmetic code book implicitly defined by a large language model.
  arXiv  Detail & Related papers  (2022-10-18T22:19:41Z)
• Diverse Human Motion Prediction via Gumbel-Softmax Sampling from an Auxiliary Space [34.83587750498361]
  Diverse human motion prediction aims at predicting multiple possible future pose sequences from a sequence of observed poses. Previous approaches usually employ deep generative networks to model the conditional distribution of data, and then randomly sample outcomes from the distribution. We propose a novel sampling strategy for sampling very diverse results from an imbalanced multimodal distribution.
  arXiv  Detail & Related papers  (2022-07-15T09:03:57Z)
• DLow: Diversifying Latent Flows for Diverse Human Motion Prediction [32.22704734791378]
  We propose a novel sampling method, Diversifying Latent Flows (DLow), to produce a diverse set of samples from a pretrained deep generative model. During training, DLow uses a diversity-promoting prior over samples as an objective to optimize the latent mappings to improve sample diversity. Our experiments demonstrate that DLow outperforms state-of-the-art baseline methods in terms of sample diversity and accuracy.
  arXiv  Detail & Related papers  (2020-03-18T17:58:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.