Are we really tilting? The mechanics of reward guidance in flow and diffusion models

• URL: http://arxiv.org/abs/2606.02884v1
• Date: Mon, 01 Jun 2026 20:56:24 GMT
• Title: Are we really tilting? The mechanics of reward guidance in flow and diffusion models
• Abstract summary: Reward guidance algorithms steer a learned generative process toward the reward-tilted measure at inference time.<n>While empirically powerful, these methods are prone to reward hacking.<n>We show that reward hacking arises from an approximation made in most practical implementations of reward-guided diffusion.
• Score: 1.4064491732635236
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reward guidance algorithms steer a learned generative process toward the reward-tilted measure at inference time. While empirically powerful, these methods are prone to reward hacking: the guided model over-optimizes the reward at the cost of fidelity to the learned distribution. Prior work has attributed this to the complexity of neural reward functions or implicit biases in diffusion training, but its fundamental origins remain poorly understood. We show that reward hacking arises from an approximation made in most practical implementations of reward-guided diffusion -- finite-particle plug-in estimation of the Doob h-function -- even in the simplest non-trivial settings of Gaussian and Gaussian mixture targets with quadratic rewards. In closed form, we isolate two distinct failure modes of the plug-in estimator: it leads to reward hacking within each mode and it cannot select high-reward modes. We propose a closed-form reward damping schedule that corrects the within-mode bias with no additional compute, and clarify the role of best-of-n sampling in compensating for the mode selection failure. Experiments on Gaussian mixture targets, a 2D checkerboard, and FLUX.1 text-to-image generation confirm that our theoretical insights carry over to practical settings.

Related papers

• GRAS: Guided Reduced-Variance Proposals and Adaptive Selection for Training-Free Reward Alignment in Discrete Diffusion [8.46069844016289]
  We study the combined regime and identify two weaknesses in how it is usually run.<n>We call the resulting method Guided Reduced-variance proposals and Adaptive Selection (GRAS)<n>GRAS is simple yet effective: across regulatory DNA and protein design it attains the best training-free reward, outperforming prior training-free methods and matching or surpassing a reward-fine-tuned model, and it remains effective even for non-differentiable rewards.
  arXiv  Detail & Related papers  (2026-08-27T03:53:59Z)
• MARBLE: Multi-Aspect Reward Balance for Diffusion RL [71.6241143519038]
  Reinforcement learning has become the dominant approach for aligning diffusion models with human preferences.<n>Existing practice deal with multiple rewards by training one specialist model per reward.<n>We propose MARBLE, a gradient-space optimization framework that maintains independent advantage estimators for each reward.
  arXiv  Detail & Related papers  (2026-05-07T16:20:42Z)
• Training-Free Adaptation of Diffusion Models via Doob's $h$-Transform [37.05492050174751]
  DOIT (Doob-Oriented Inference-time Transformation) is a training-free and computationally efficient adaptation method.<n>We leverage Doob's $h$-transform to realize this transport, which induces a dynamic correction to the diffusion sampling process.<n>Our method consistently outperforms state-of-the-art baselines while preserving sampling efficiency.
  arXiv  Detail & Related papers  (2026-02-18T05:44:19Z)
• FAIL: Flow Matching Adversarial Imitation Learning for Image Generation [52.643484089126844]
  Post-training of flow matching models-aligning the output distribution with a high-quality target-is mathematically equivalent to Imitation learning.<n>We propose Flow Matching Adrial Learning (FAIL), which minimizes policy-expert divergence through adversarial training without explicit rewards or pairwise comparisons.
  arXiv  Detail & Related papers  (2026-02-12T16:36:33Z)
• GDRO: Group-level Reward Post-training Suitable for Diffusion Models [55.948229011478304]
  Group-level rewards successfully align the model with the targeted reward.<n>Group-level Direct Reward Optimization (GDRO) is a new post-training paradigm for group-level reward alignment.<n>GDRO supports full offline training that saves the large time cost for image rollout sampling.<n>It is diffusion-sampler-independent, which eliminates the need for the ODE-to-SDE approximation to obtainity.
  arXiv  Detail & Related papers  (2026-01-05T11:47:18Z)
• Test-time scaling of diffusions with flow maps [68.79792714591564]
  A common recipe to improve diffusion models at test-time is to introduce the gradient of the reward into the dynamics of the diffusion itself.<n>We propose a simple solution by working directly with a flow map.<n>By exploiting a relationship between the flow map and velocity field governing the instantaneous transport, we construct an algorithm, Flow Map Trajectory Tilting (FMTT), which provably performs better ascent on the reward than standard test-time methods.
  arXiv  Detail & Related papers  (2025-11-27T18:44:12Z)
• Conditioning Diffusions Using Malliavin Calculus [18.62300657866048]
  In generative modelling and optimal control, a central computational task is to modify a reference diffusion process to maximise a given terminal-time reward.<n>We introduce a novel framework based on Malliavin calculus and centred around a generalisation of the Tweedie score formula to nonlinear differential equations.<n>This allows our approach to handle a broad range of applications, like diffusion bridges, or adding conditional controls to an already trained diffusion model.
  arXiv  Detail & Related papers  (2025-04-04T14:10:21Z)
• RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution [50.171320156632866]
  Reinforcement learning from human feedback offers a promising approach to aligning large language models with human preferences.<n>Current reward models operate as sequence-to-one models, allocating a single, sparse, and delayed reward to an entire output sequence.<n>We propose a more fine-grained, token-level guidance approach for RL training.
  arXiv  Detail & Related papers  (2024-11-13T02:45:21Z)
• Rejection via Learning Density Ratios [50.91522897152437]
  Classification with rejection emerges as a learning paradigm which allows models to abstain from making predictions.<n>We propose a different distributional perspective, where we seek to find an idealized data distribution which maximizes a pretrained model's performance.<n>Our framework is tested empirically over clean and noisy datasets.
  arXiv  Detail & Related papers  (2024-05-29T01:32:17Z)
• Extracting Reward Functions from Diffusion Models [7.834479563217133]
  Decision-making diffusion models can be trained on lower-quality data, and then be steered with a reward function to generate near-optimal trajectories. We consider the problem of extracting a reward function by comparing a decision-making diffusion model that models low-reward behavior and one that models high-reward behavior. We show that our approach generalizes beyond sequential decision-making by learning a reward-like function from two large-scale image generation diffusion models.
  arXiv  Detail & Related papers  (2023-06-01T17:59:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.