FuguReport
Daily Report / 2026-09-27

Daily Paper Briefing

2026-09-27 1 Paper

Diffusion Reward Models

Authors Xiangyang Wang, Bingxiang He, Zeyuan Liu, Jiaze WangZiqing Qiao, Yuxin Zuo, Huan-ang Gao, Cheng Qian, Wenbin Zhang, Ran Li, Youbang Sun, Ning Ding, Yuanchun Shi, Zhiyuan Liu, Chaojun Xiao, Chun Yu
Affiliations Tsinghua University / University of Illinois Urbana-Champaign
Categories Method / Reward Models / Conditional density reward estimation, Evaluation / Model Evaluation / Benchmark comparison on multimodal rewards, Application / Multimodal Learning / Multimodal reward structure recovery
License CC BY 4.0

Abstract Overview

This paper proposes Diffusion Reward Models (DRM), which recast reward modeling as conditional density estimation over rewards given a prompt-response pair rather than predicting a single scalar score. The method uses a frozen LLM encoder and a lightweight diffusion transformer head to model reward distributions without assuming a fixed parametric family, and the same framework supports both multi-attribute regression and pairwise preference training. At inference, DRM produces multiple reward samples that can be summarized into means, variances, or quantiles, enabling both standard scalar scoring and distribution-aware decision rules. The evaluation spans five public reward-model benchmarks, analyses of human disagreement, and downstream RLHF experiments.

Novelty

The main novelty is replacing conventional scalar or fixed-family parametric reward heads with a diffusion-based head that models the full conditional reward distribution non-parametrically. The framework provides a unified architecture for both multi-attribute regression and pairwise preference data while introducing a reward-axis test-time scaling mechanism via repeated diffusion sampling.

Results

Under matched data and backbone, DRM-Multi-8B achieves a 66.2 average benchmark score, outperforming ArmoRM (62.3) and QRM-Llama3.1-8B-v2 (64.1). In distributional evaluations, DRM aligns more closely with human rating distributions than baselines, with uncertainty-aware rejection and lower-confidence-bound aggregation improving decision accuracy. When used as the training reward in downstream RLHF, DRM-Multi improves policy performance on Arena-Hard v2 from 1.3 to 2.0 and MT-Bench from 73.6 to 74.8 over the FsfairX baseline.

Key Points

  1. DRM models reward as a non-parametric conditional distribution using a Diffusion Transformer head on top of a frozen LLM encoder.
  2. The architecture supports both multi-attribute supervision and pairwise preference objectives while enabling reward-axis test-time scaling through multiple reverse diffusion samples.
  3. Evaluations across five RM benchmarks, human disagreement analyses, and downstream RLHF demonstrate superior performance over matched baselines and effective risk-sensitive aggregation.

References

GitHub Hugging Face
This page was created using generative AI such as GPT-5, Claude Opus 4, Gemini 3, Gemini 3.1 Flash Image, and their higher-end successor versions. No guarantee can be made regarding its contents.