Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization

• URL: http://arxiv.org/abs/2604.07343v1
• Date: Wed, 08 Apr 2026 17:55:00 GMT
• Title: Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
• Abstract summary: We introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences.<n>We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual.<n>Our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines.
• Score: 12.964518425036168
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values. While benchmarks for general response quality are prevalent, evaluating how well reward models account for individual user preferences remains an open challenge. To bridge this gap, we introduce Personalized RewardBench, a novel benchmark designed to rigorously assess reward models' capacity to model personalized preferences. We construct chosen and rejected response pairs based on strict adherence to (or violation of) user-specific rubrics, ensuring that preference distinctions are uniquely tailored to the individual. In particular, human evaluations confirm that the primary discriminative factor between pairs is strictly personal preference, with both responses maintaining high general quality (e.g., correctness, relevance and helpfulness). Extensive testing reveals that existing state-of-the-art reward models struggle significantly with personalization, peaking at an accuracy of just 75.94%. Crucially, because an effective reward model benchmark should predict a reward model's performance on downstream tasks, we conduct experiments demonstrating that our benchmark exhibits a significantly higher correlation with downstream performance in both Best-of-N (BoN) sampling and Proximal Policy Optimization (PPO) compared to existing baselines. These findings establish Personalized RewardBench as a robust and accurate proxy for evaluating reward models' performance in downstream applications.

Related papers

• Dynamically Allocating Evaluation Effort for Model Ranking [60.00410111594536]
  We formalize multi-model human evaluation as a best-arm identification problem in a multi-armed bandit setup with correlated arms.<n>We show that it improves discrimination between top-performing models.<n>This makes evaluations faster, cheaper and more aligned with large-scale competition evaluation goals.
  arXiv  Detail & Related papers  (2026-08-04T10:33:04Z)
• CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modeling [61.75914342638658]
  CDRRM (Contrast-Driven Reward Model) is a framework for high-quality rubric generation and guided preference judgment.<n>This work offers a scalable, interpretable, and data-efficient path for reward modeling.
  arXiv  Detail & Related papers  (2026-03-09T07:15:23Z)
• Learning Ordinal Probabilistic Reward from Preferences [25.069054134899744]
  We introduce a novel reward modeling paradigm: Probabilistic Reward Model (PRM)<n>Instead of modeling reward as a deterministic scalar, our approach treats it as a random variable, learning a full probability distribution for the quality of each response.<n>Building on OPRM, we propose a data-efficient training strategy called Region Flooding Tuning (RgFT)
  arXiv  Detail & Related papers  (2026-02-13T06:43:02Z)
• P-Check: Advancing Personalized Reward Model via Learning to Generate Dynamic Checklist [11.399221632873934]
  We propose P-Check, a novel personalized reward modeling framework.<n>P-Check trains a plug-and-play checklist generator that synthesizes dynamic evaluation criteria for guiding the reward prediction.<n>We conduct experiments and demonstrate that P-Check not only improves reward accuracy but also enhances downstream personalized generation.
  arXiv  Detail & Related papers  (2026-01-06T12:53:53Z)
• Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models [63.00458229517523]
  This work addresses the evaluation challenge of reward models by probing preference representations.<n>We construct a Multi-dimensional Reward Model Benchmark (MRMBench), a collection of six probing tasks for different preference dimensions.<n>We introduce an analysis method, inference-time probing, which identifies the dimensions used during the reward prediction and enhances its interpretability.
  arXiv  Detail & Related papers  (2025-11-16T05:29:29Z)
• RewardBench 2: Advancing Reward Model Evaluation [71.65938693914153]
  Reward models are used throughout the post-training of language models to capture nuanced signals from preference data.<n>The community has begun establishing best practices for evaluating reward models.<n>This paper introduces RewardBench 2, a new multi-skill reward modeling benchmark.
  arXiv  Detail & Related papers  (2025-06-02T17:54:04Z)
• Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization [15.729285736811383]
  Reward models play a crucial role in reinforcement learning from human feedback.<n>Existing benchmarks for reward models show a weak correlation with the performance of optimized policies.
  arXiv  Detail & Related papers  (2025-05-19T06:43:08Z)
• Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment [51.14207112118503]
  We introduce preference embedding, an approach that embeds responses into a latent space to capture preferences efficiently.<n>We also propose preference score-based General Preference Optimization (GPO), which generalizes reward-based reinforcement learning from human feedback.<n>Our method may enhance the alignment of foundation models with nuanced human values.
  arXiv  Detail & Related papers  (2024-10-03T04:22:55Z)
• Evaluating Robustness of Reward Models for Mathematical Reasoning [14.97819343313859]
  We introduce a new design for reliable evaluation of reward models, and to validate this, we construct RewardMATH. We demonstrate that the scores on RewardMATH strongly correlate with the results of optimized policy and effectively estimate reward overoptimization.
  arXiv  Detail & Related papers  (2024-10-02T16:39:58Z)
• RewardBench: Evaluating Reward Models for Language Modeling [100.28366840977966]
  We present RewardBench, a benchmark dataset and code-base for evaluation of reward models. The dataset is a collection of prompt-chosen-rejected trios spanning chat, reasoning, and safety. On the RewardBench leaderboard, we evaluate reward models trained with a variety of methods.
  arXiv  Detail & Related papers  (2024-03-20T17:49:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.