Variance-aware Reward Modeling with Anchor Guidance

• URL: http://arxiv.org/abs/2605.11865v1
• Date: Tue, 12 May 2026 09:46:26 GMT
• Title: Variance-aware Reward Modeling with Anchor Guidance
• Abstract summary: We propose Anchor-guided Variance-aware Reward Modeling, a framework that resolves non-identifiability.<n>Across simulation studies and four real-world diverging-preference datasets, our method consistently improves reward modeling performance.
• Score: 10.561814492691534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Standard Bradley--Terry (BT) reward models are limited when human preferences are pluralistic. Although soft preference labels preserve disagreement information, BT can only express it by shrinking reward margins. Gaussian reward models provide an alternative by jointly predicting a reward mean and a reward variance, but suffer from a fundamental non-identifiability from pairwise preferences alone. We propose Anchor-guided Variance-aware Reward Modeling, a framework that resolves this non-identifiability by augmenting preference data with two coarse response-level anchor labels. Building on this, we prove that two anchors are sufficient for identification, develop a joint training objective and establish a non-asymptotic convergence rate for both the estimated reward mean and variance functions. Across simulation studies and four real-world diverging-preference datasets, our method consistently improves reward modeling performance and downstream RLHF, including PPO training and best-of-$N$ selection.

Related papers

• UBP2: Uncertainty-Balanced Preference Planning for Efficient Preference-based Reinforcement Learning [4.276809570692494]
  We introduce a model-based approach that actively directs exploration by jointly reasoning over uncertainties in the reward, dynamics, and value functions.<n>Our method, Uncertainty-Balanced Preference Planning (UBP2), uses ensembles of reward, dynamics, and value function models to evaluate candidate trajectories.<n>Under standard assumptions, we establish sublinear regret guarantees for both finite-horizon and infinite-horizon settings.
  arXiv  Detail & Related papers  (2026-06-17T17:54:32Z)
• Beyond Pairwise Preferences: Listwise Reward-Aware Alignment for Diffusion Models [73.08789211016567]
  Diffusion LAIR is a reward-aware listwise preference optimization method for diffusion models.<n>Experiments show it outperforms strong preference optimization baselines on text-to-image generation, compositional generation, and image editing benchmarks.
  arXiv  Detail & Related papers  (2026-05-26T03:09:24Z)
• ImplicitRM: Unbiased Reward Modeling from Implicit Preference Data for LLM alignment [49.93516092785756]
  Current reward modeling is heavily contingent upon experimental feedback data with high collection costs.<n>ImplicitRM aims to learn unbiased reward models from implicit preference data.<n> Experiments demonstrate that ImplicitRM learns accurate reward models across implicit preference datasets.
  arXiv  Detail & Related papers  (2026-03-24T13:32:14Z)
• Multi-dimensional Preference Alignment by Conditioning Reward Itself [32.33870784484853]
  Multi Reward Conditional DPO resolves reward conflicts by introducing a disentangled Bradley-Terry objective.<n>Experiments on Stable 1.5 and SDXL demonstrate that MCDPO achieves superior performance on benchmarks.
  arXiv  Detail & Related papers  (2025-12-11T02:44:31Z)
• MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resources [113.33902847941941]
  Variance-Aware Sampling (VAS) is a data selection strategy guided by Variance Promotion Score (VPS)<n>We release large-scale, carefully curated resources containing 1.6M long CoT cold-start data and 15k RL QA pairs.<n> Experiments across mathematical reasoning benchmarks demonstrate the effectiveness of both the curated data and the proposed VAS.
  arXiv  Detail & Related papers  (2025-09-25T14:58:29Z)
• Reward Models Can Improve Themselves: Reward-Guided Adversarial Failure Mode Discovery for Robust Reward Modeling [27.11560841914813]
  We introduce REFORM, a self-improving reward modeling framework that enhances robustness by using the reward model itself to guide the generation of falsely scored responses.<n>We evaluate REFORM on two widely used preference datasets Anthropic Helpful Harmless (HH) and PKU Beavertails.
  arXiv  Detail & Related papers  (2025-07-08T21:56:33Z)
• Calibrated Multi-Preference Optimization for Aligning Diffusion Models [90.15024547673785]
  Calibrated Preference Optimization (CaPO) is a novel method to align text-to-image (T2I) diffusion models.<n>CaPO incorporates the general preference from multiple reward models without human annotated data.<n> Experimental results show that CaPO consistently outperforms prior methods.
  arXiv  Detail & Related papers  (2025-02-04T18:59:23Z)
• Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives [14.401557416713315]
  We revisit the foundations of using Bradley-Terry (BT) models in reward modeling.<n>We argue that the BT model is not a necessary choice from the perspective of downstream optimization.<n>We propose a simple and straightforward upper-bound algorithm, compatible with off-the-shelf binary classifiers.
  arXiv  Detail & Related papers  (2024-11-07T18:57:03Z)
• Robust Preference Optimization through Reward Model Distillation [68.65844394615702]
  Direct Preference Optimization (DPO) is a popular offline alignment method that trains a policy directly on preference data.<n>We analyze this phenomenon and use distillation to get a better proxy for the true preference distribution over generation pairs.<n>Our results show that distilling from such a family of reward models leads to improved robustness to distribution shift in preference annotations.
  arXiv  Detail & Related papers  (2024-05-29T17:39:48Z)
• Secrets of RLHF in Large Language Models Part II: Reward Modeling [134.97964938009588]
  We introduce a series of novel methods to mitigate the influence of incorrect and ambiguous preferences in the dataset. We also introduce contrastive learning to enhance the ability of reward models to distinguish between chosen and rejected responses.
  arXiv  Detail & Related papers  (2024-01-11T17:56:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.