Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown

• URL: http://arxiv.org/abs/2410.00847v2
• Date: Wed, 12 Feb 2025 03:34:29 GMT
• Title: Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown
• Authors: Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, Junge Zhang,
• Abstract summary: We introduce the Uncertainty-aware Reward Model (URM) and its ensemble variant, URME.<n>URM employs a probabilistic value head to capture aleatoric uncertainty by modeling the distribution of disentangled human preference attributes.<n> URME further quantifies uncertainty by examining discrepancies among individual URMs within the ensemble, enabling identification of unreliable evaluations.
• Score: 20.753374166695494
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reward models (RMs) are essential for aligning large language models (LLM) with human expectations. However, existing RMs struggle to capture the stochastic and uncertain nature of human preferences and fail to assess the reliability of reward predictions. To address these challenges, we introduce the Uncertainty-aware Reward Model (URM) and its ensemble variant, URME. URM employs a probabilistic value head to capture aleatoric uncertainty by modeling the distribution of disentangled human preference attributes. URME further quantifies epistemic uncertainty by examining discrepancies among individual URMs within the ensemble, enabling identification of unreliable evaluations. Our empirical evaluations demonstrate that URM achieves strong performance on RewardBench, outperforming competitive large-scale models. Additionally, extensive experiments, including best-of-n sampling (BoN), iterative direct preference optimization (iterative DPO), and proximal policy optimization (PPO), demonstrate that URM and URME significantly enhance LLMs' generation quality. Notably, reward predictions with lower uncertainty are far more reliable, demonstrate significantly higher quality, and result in substantially improved alignment.

Related papers

• Establishing Reliability Metrics for Reward Models in Large Language Models [17.26528659228218]
  The reward model (RM) that represents human preferences plays a crucial role in optimizing the outputs of large language models (LLMs) We propose the itunderlineReliable at underline$eta$ (RETA) metric to measure the reliability of RMs. On top of RETA, we present an integrated benchmarking pipeline that allows anyone to evaluate their own RM without incurring additional Oracle labeling costs.
  arXiv  Detail & Related papers  (2025-04-21T03:39:33Z)
• Energy-Based Reward Models for Robust Language Model Alignment [9.843359827321194]
  We introduce Energy-Based Reward Model (EBRM), a lightweight post-hoc refinement framework for Reward Models (RMs) EBRM models the reward distribution explicitly, capturing uncertainty in human preferences and mitigating the impact of noisy or misaligned annotations. Empirical evaluations demonstrate significant improvements in robustness and generalization, achieving up to a 5.97% improvement in safety-critical alignment tasks.
  arXiv  Detail & Related papers  (2025-04-17T17:47:15Z)
• Adversarial Training of Reward Models [74.17196154247964]
  We introduce Adv-RM, a novel adversarial training framework that automatically identifies adversarial examples. By leveraging reinforcement learning, Adv-RM trains a policy to expose vulnerabilities in large state-of-the-art reward models. We demonstrate that Adv-RM significantly outperforms conventional reward training.
  arXiv  Detail & Related papers  (2025-04-08T15:38:25Z)
• Uncertainty-Aware Step-wise Verification with Generative Reward Models [42.17917357636397]
  We propose leveraging uncertainty quantification (UQ) to enhance the reliability of step-wise verification with generative reward models. We introduce CoT Entropy, a novel UQ method that outperforms existing approaches in quantifying a PRM's uncertainty in step-wise verification.
  arXiv  Detail & Related papers  (2025-02-16T20:00:56Z)
• The Lessons of Developing Process Reward Models in Mathematical Reasoning [62.165534879284735]
  Process Reward Models (PRMs) aim to identify and mitigate intermediate errors in the reasoning processes. We develop a consensus filtering mechanism that effectively integrates Monte Carlo (MC) estimation with Large Language Models (LLMs) We release a new state-of-the-art PRM that outperforms existing open-source alternatives.
  arXiv  Detail & Related papers  (2025-01-13T13:10:16Z)
• Reward-Robust RLHF in LLMs [25.31456438114974]
  Large Language Models (LLMs) continue to progress toward more advanced forms of intelligence. The reliance on reward-model-based (RM-based) alignment methods introduces significant challenges. We introduce a reward-robust RLHF framework aimed at addressing these fundamental challenges.
  arXiv  Detail & Related papers  (2024-09-18T02:35:41Z)
• Self-Evolutionary Large Language Models through Uncertainty-Enhanced Preference Optimization [9.618391485742968]
  Iterative preference optimization has recently become one of the de-facto training paradigms for large language models (LLMs) We present an uncertainty-enhanced textbfPreference textbfOptimization framework to make the LLM self-evolve with reliable feedback. Our framework substantially alleviates the noisy problem and improves the performance of iterative preference optimization.
  arXiv  Detail & Related papers  (2024-09-17T14:05:58Z)
• Semi-Supervised Reward Modeling via Iterative Self-Training [52.48668920483908]
  We propose Semi-Supervised Reward Modeling (SSRM), an approach that enhances RM training using unlabeled data. We demonstrate that SSRM significantly improves reward models without incurring additional labeling costs. Overall, SSRM substantially reduces the dependency on large volumes of human-annotated data, thereby decreasing the overall cost and time involved in training effective reward models.
  arXiv  Detail & Related papers  (2024-09-10T22:57:58Z)
• Beyond RMSE and MAE: Introducing EAUC to unmask hidden bias and unfairness in dyadic regression models [5.336076422485076]
  We show that non-uniformity in the observed value distributions of individual entities leads to severely biased predictions in state-of-the-art models. We introduce Eccentricity-Area Under the Curve (EAUC) as a new metric that can quantify it in all studied models and datasets.
  arXiv  Detail & Related papers  (2024-01-19T13:41:08Z)
• Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization [59.758009422067]
  We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. We propose a new uncertainty Bellman equation (UBE) whose solution converges to the true posterior variance over values. We introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC) that can be applied for either risk-seeking or risk-averse policy optimization.
  arXiv  Detail & Related papers  (2023-12-07T15:55:58Z)
• Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation [50.920911532133154]
  The intrinsic ill-posedness and ordinal-sensitive nature of monocular depth estimation (MDE) models pose major challenges to the estimation of uncertainty degree. We propose to model the uncertainty of MDE models from the perspective of the inherent probability distributions. By simply introducing additional training regularization terms, our model, with surprisingly simple formations and without requiring extra modules or multiple inferences, can provide uncertainty estimations with state-of-the-art reliability.
  arXiv  Detail & Related papers  (2023-07-19T12:11:15Z)
• Training, Architecture, and Prior for Deterministic Uncertainty Methods [33.45069308137142]
  This work investigates important design choices in Deterministic Uncertainty Methods (DUMs) We show that training schemes decoupling the core architecture and the uncertainty head schemes can significantly improve uncertainty performances. Contrary to other Bayesian models, we show that the prior defined by DUMs do not have a strong effect on the final performances.
  arXiv  Detail & Related papers  (2023-03-10T09:00:52Z)
• Rethinking Missing Data: Aleatoric Uncertainty-Aware Recommendation [59.500347564280204]
  We propose a new Aleatoric Uncertainty-aware Recommendation (AUR) framework. AUR consists of a new uncertainty estimator along with a normal recommender model. As the chance of mislabeling reflects the potential of a pair, AUR makes recommendations according to the uncertainty.
  arXiv  Detail & Related papers  (2022-09-22T04:32:51Z)
• Uncertainty-Driven Action Quality Assessment [67.20617610820857]
  We propose a novel probabilistic model, named Uncertainty-Driven AQA (UD-AQA), to capture the diversity among multiple judge scores. We generate the estimation of uncertainty for each prediction, which is employed to re-weight AQA regression loss. Our proposed method achieves competitive results on three benchmarks including the Olympic events MTL-AQA and FineDiving, and the surgical skill JIGSAWS datasets.
  arXiv  Detail & Related papers  (2022-07-29T07:21:15Z)
• Policy Gradient Bayesian Robust Optimization for Imitation Learning [49.881386773269746]
  We derive a novel policy gradient-style robust optimization approach, PG-BROIL, to balance expected performance and risk. Results suggest PG-BROIL can produce a family of behaviors ranging from risk-neutral to risk-averse.
  arXiv  Detail & Related papers  (2021-06-11T16:49:15Z)
• Approaching Neural Network Uncertainty Realism [53.308409014122816]
  Quantifying or at least upper-bounding uncertainties is vital for safety-critical systems such as autonomous vehicles. We evaluate uncertainty realism -- a strict quality criterion -- with a Mahalanobis distance-based statistical test. We adopt it to the automotive domain and show that it significantly improves uncertainty realism compared to a plain encoder-decoder model.
  arXiv  Detail & Related papers  (2021-01-08T11:56:12Z)
• On the model-based stochastic value gradient for continuous reinforcement learning [50.085645237597056]
  We show that simple model-based agents can outperform state-of-the-art model-free agents in terms of both sample-efficiency and final reward. Our findings suggest that model-based policy evaluation deserves closer attention.
  arXiv  Detail & Related papers  (2020-08-28T17:58:29Z)
• Providing reliability in Recommender Systems through Bernoulli Matrix Factorization [63.732639864601914]
  This paper proposes Bernoulli Matrix Factorization (BeMF) to provide both prediction values and reliability values. BeMF acts on model-based collaborative filtering rather than on memory-based filtering. The more reliable a prediction is, the less liable it is to be wrong.
  arXiv  Detail & Related papers  (2020-06-05T14:24:27Z)
• Model Uncertainty Quantification for Reliable Deep Vision Structural Health Monitoring [2.5126058470073263]
  This paper proposes Bayesian inference for deep vision structural health monitoring models. Uncertainty can be quantified using the Monte Carlo dropout sampling. Three independent case studies for cracks, local damage identification, and bridge component detection are investigated.
  arXiv  Detail & Related papers  (2020-04-10T17:54:10Z)
• Uncertainty-Gated Stochastic Sequential Model for EHR Mortality Prediction [6.170898159041278]
  We present a novel variational recurrent network that estimates the distribution of missing variables, updates hidden states, and predicts the possibility of in-hospital mortality. It is noteworthy that our model can conduct these procedures in a single stream and learn all network parameters jointly in an end-to-end manner.
  arXiv  Detail & Related papers  (2020-03-02T04:41:28Z)
• Learning to Predict Error for MRI Reconstruction [67.76632988696943]
  We demonstrate that predictive uncertainty estimated by the current methods does not highly correlate with prediction error. We propose a novel method that estimates the target labels and magnitude of the prediction error in two steps.
  arXiv  Detail & Related papers  (2020-02-13T15:55:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.