2026-07-02 Daily Report: Optimizing Visual Generative Models via Distribution-wise Rewards

Optimizing Visual Generative Models via Distribution-wise Rewards

Authors Ruihang Li, Mengde Xu, Shuyang Gu, Leigang Qu, Fuli Feng, Han Hu, Wenjie Wang

Affiliations University of Science and Technology of China / Shanghai AI Laboratory / Huawei / Shanghai Innovation Institute

Categories Method / Reward Optimization / Distribution-based reward framework, Evaluation / Model Evaluation / FID metric improvement analysis, Application / Generative Modeling / Visual generative model enhancement

License CC BY-SA 4.0

Abstract Overview

This paper proposes a reinforcement-learning framework for visual generative models that uses distribution-wise rewards rather than conventional sample-wise rewards. The motivation is that sample-wise rewards can encourage reward hacking, leading to artifacts and reduced diversity, whereas distribution-level signals better reflect alignment with the real image distribution. To make such rewards tractable, the authors introduce a subset-replace strategy that estimates FID-based rewards by updating only a small subset of a generated reference set during rollout. They further apply this subset-replace reward strategy to optimize post-hoc model merging coefficients, aiming to avoid the train-inference inconsistency associated with SDE-based RL training.

Novelty

The main novelty is the utilization of distribution-wise reward signals, instantiated with FID, for the reinforcement optimization of visual generative models. A second distinctive contribution is the subset-replace mechanism for obtaining dense, lower-cost reward estimates, alongside utilizing RL to optimize post-hoc model merging coefficients under ODE-based sampling.

Results

On ImageNet 256x256, the method improves the SiT-XL/2 baseline from an FID-50K of 8.30 to 5.77 via RL, and reduces the FD_DINOv2 score from 230.39 to 164.88; a rejection-sampling variant reaches an FID of 6.98. For post-hoc model merging on ImageNet 512x512, the RL-based optimization improves EDM2-XS from an FID of 3.74 to 3.52 and EDM2-S from 2.57 to 2.52.

Key Points

Sample-wise rewards in image-generation RL are prone to reward hacking, which degrades diversity and introduces visual artifacts.
A novel subset-replace strategy approximates distribution-wise rewards by replacing small subsets within a generated reference set, resolving the extreme computational cost and feedback sparsity of FID-based rewards.
The proposed framework is successfully applied both for direct RL fine-tuning and for RL-based post-hoc model merging, demonstrating consistent quantitative improvements across SiT and EDM2 architectures.

References

arXiv: https://arxiv.org/abs/2607.02291v1
Fugu-MT: https://fugumt.com/fugumt/paper_check/2607.02291v1
Hugging Face Papers: https://huggingface.co/papers/2607.02291