GRAS: Guided Reduced-Variance Proposals and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion

• URL: http://arxiv.org/abs/2608.26585v1
• Date: Thu, 27 Aug 2026 03:53:59 GMT
• Title: GRAS: Guided Reduced-Variance Proposals and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion
• Abstract summary: We study the combined regime and identify two weaknesses in how it is usually run.<n>We call the resulting method Guided Reduced-variance proposals and Adaptive Selection (GRAS)<n>GRAS is simple yet effective: across regulatory DNA and protein design it attains the best training-free reward, outperforming prior training-free methods and matching or surpassing a reward-fine-tuned model, and it remains effective even for non-differentiable rewards.
• Score: 8.46069844016289
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Discrete diffusion models have become a strong, widely adopted class of generators for sequence data, and steering them toward a downstream reward at inference time, without any retraining, is increasingly important. Such training-free steering is done by gradient guidance, by search, or by combining the two. We study the combined regime and identify two weaknesses in how it is usually run: the guided proposal estimates its gradient from a single noisy sample, and the search then resamples particles at a fixed temperature that ignores how rewards spread across each denoising step. We address both with a small set of changes that add no denoiser cost. For the proposal, we lower the estimator variance with a Rao-Blackwellized reveal for differentiable rewards and a leave-one-out baseline for non-differentiable ones; for the search, we standardize the per-step values into a group-relative advantage and prove it collapses to a single active ingredient, an adaptive resampling temperature. We call the resulting method Guided Reduced-variance proposals and Adaptive Selection (GRAS). GRAS is simple yet effective: across regulatory DNA and protein design it attains the best training-free reward, outperforming prior training-free methods and matching or surpassing a reward-fine-tuned model, and it remains effective even for non-differentiable rewards.

Related papers

• Simplex Relaxation for Discrete Diffusion [58.33696160279345]
  We introduce Simplax, an exact Dirichlet--categorical augmentation that couples each corrupted categorical state with an auxiliary simplex-valued variable.<n>Simplax improves the generative perplexity--entropy tradeoff on unconditional OpenWebText generation.
  arXiv  Detail & Related papers  (2026-08-11T07:58:17Z)
• Latent Reward Registers for Diffusion Preference Alignment [46.49072106092868]
  We propose Latent Reward Registers, a mechanism that estimates terminal preference directly from intermediate noisy latents.<n>This independent readout mechanism extracts latent reward evidence without altering the generator's hidden states or velocity field.<n>For training, Reward-Gradient On-Policy Distillation (RG-OPD) distills reward-guided updates along on-policy trajectories, bypassing the computationally expensive rollouts of standard policy gradients.<n>For inference, Reward-Guided Sampling (RGS) steers trajectories via magnitude-matched reward gradients without parameter updates.
  arXiv  Detail & Related papers  (2026-08-04T17:00:52Z)
• Diversify Diffusion with Temperature Sampling and Variance-Corrective Time Shifting [57.75750707482832]
  We introduce variance-corrective time shifting, a training-free fix that queries the network at a shifted timestep and scales the resulting score by $$, canceling the variance inflation while preserving the mode reweighting.<n>We demonstrate consistent gains at minimal cost to sample quality and condition fidelity across DiT, Stable Diffusion and Motion Diffusion models.
  arXiv  Detail & Related papers  (2026-07-12T17:32:29Z)
• Are we really tilting? The mechanics of reward guidance in flow and diffusion models [1.4064491732635236]
  Reward guidance algorithms steer a learned generative process toward the reward-tilted measure at inference time.<n>While empirically powerful, these methods are prone to reward hacking.<n>We show that reward hacking arises from an approximation made in most practical implementations of reward-guided diffusion.
  arXiv  Detail & Related papers  (2026-06-01T20:56:24Z)
• Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models [73.08789211016567]
  Diffusion LAIR is a reward-aware listwise preference optimization method for diffusion models.<n>Experiments show it outperforms strong preference optimization baselines on text-to-image generation, compositional generation, and image editing benchmarks.
  arXiv  Detail & Related papers  (2026-05-26T03:09:24Z)
• MARBLE: Multi-Aspect Reward Balance for Diffusion RL [71.6241143519038]
  Reinforcement learning has become the dominant approach for aligning diffusion models with human preferences.<n>Existing practice deal with multiple rewards by training one specialist model per reward.<n>We propose MARBLE, a gradient-space optimization framework that maintains independent advantage estimators for each reward.
  arXiv  Detail & Related papers  (2026-05-07T16:20:42Z)
• Scaling Adversarial Training via Data Selection [0.0]
  We propose emphSelective Adversa Training, which perturbs only a subset of critical samples in each minibatch.<n>Experiments on MNIST and CIFAR-10 show that the proposed methods achieve robustness comparable to, or even exceeding, full PGD adversarial training.
  arXiv  Detail & Related papers  (2025-12-26T15:50:33Z)
• Bridging the Gap: Addressing Discrepancies in Diffusion Model Training for Classifier-Free Guidance [1.6804613362826175]
  Diffusion models have emerged as a pivotal advancement in generative models. In this paper we aim to underscore a discrepancy between conventional training methods and the desired conditional sampling behavior. We introduce an updated loss function that better aligns training objectives with sampling behaviors.
  arXiv  Detail & Related papers  (2023-11-02T02:03:12Z)
• Data Pruning via Moving-one-Sample-out [61.45441981346064]
  We propose a novel data-pruning approach called moving-one-sample-out (MoSo) MoSo aims to identify and remove the least informative samples from the training set. Experimental results demonstrate that MoSo effectively mitigates severe performance degradation at high pruning ratios.
  arXiv  Detail & Related papers  (2023-10-23T08:00:03Z)
• Jo-SRC: A Contrastive Approach for Combating Noisy Labels [58.867237220886885]
  We propose a noise-robust approach named Jo-SRC (Joint Sample Selection and Model Regularization based on Consistency) Specifically, we train the network in a contrastive learning manner. Predictions from two different views of each sample are used to estimate its "likelihood" of being clean or out-of-distribution.
  arXiv  Detail & Related papers  (2021-03-24T07:26:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.