When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient

• URL: http://arxiv.org/abs/2604.25872v1
• Date: Tue, 28 Apr 2026 17:10:15 GMT
• Title: When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradient
• Abstract summary: Training language models via reinforcement learning often relies on imperfect proxy rewards.<n>Standard metrics for assessing the quality of proxy rewards, such as ranking accuracy, treat incorrect rewards as strictly harmful.<n>We categorize reward errors according to their effect on the increase in ground truth reward.
• Score: 28.276272279321578
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training language models via reinforcement learning often relies on imperfect proxy rewards, since ground truth rewards that precisely define the intended behavior are rarely available. Standard metrics for assessing the quality of proxy rewards, such as ranking accuracy, treat incorrect rewards as strictly harmful. In this work, however, we highlight that not all deviations from the ground truth are equal. By theoretically analyzing which outputs attract probability during policy gradient optimization, we categorize reward errors according to their effect on the increase in ground truth reward. The analysis establishes that reward errors, though conventionally viewed as harmful, can also be benign or even beneficial by preventing the policy from stalling around outputs with mediocre ground truth reward. We then present two practical implications of our theory. First, for reinforcement learning from human feedback (RLHF), we develop reward model evaluation metrics that account for the harmfulness of reward errors. Compared to standard ranking accuracy, these metrics typically correlate better with the performance of a language model after RLHF, yet gaps remain in robustly evaluating reward models. Second, we provide insights for reward design in settings with verifiable rewards. A key theme underlying our results is that the effectiveness of a proxy reward function depends heavily on its interaction with the initial policy and learning algorithm.

Related papers

• Counterfactual Shapley Credit Assignment [49.39649767581182]
  Credit Assignment Problem (CAP) is fundamental to developing efficient and explainable Reinforcement Learning (RL) agents.<n>We introduce Counterfactual Shapley Credit Assignment, a novel framework that attributes credit and blame via the Counterfactual Shapley ($$-value)<n>We derive a consistent estimator that computes $$$-values, enabling a new class of policy methods, $$-PPO, combined with Prioritized Trajectory Replay (PTR)
  arXiv  Detail & Related papers  (2026-07-18T23:28:20Z)
• Discretizing Reward Models [70.71071807050916]
  We show that many popular reward models are oversensitive, assigning different scores to equally good responses.<n>We propose evaluating reward models using measures of "discriminative ability" and "specificity"
  arXiv  Detail & Related papers  (2026-06-19T23:13:59Z)
• VIMPO: Value-Implicit Policy Optimization for LLMs [106.88933849641272]
  Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token.<n>Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability.<n>We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from optimality conditions of KL-regularized reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-18T09:44:12Z)
• Factored Causal Representation Learning for Robust Reward Modeling in RLHF [40.483487519518896]
  A reliable reward model is essential for aligning large language models with human preferences.<n>Standard reward models are susceptible to spurious features that are not causally related to human labels.<n>This can lead to reward hacking, where high predicted reward does not translate into better behavior.
  arXiv  Detail & Related papers  (2026-01-29T07:18:45Z)
• Probabilistic Uncertain Reward Model [27.40414952747553]
  This paper proposes the Probabilistic Uncertain Reward Model (PURM) to learn the reward distributions that emerged from the preference data.<n>We show that PURM outperforms existing methods with more accurate reward and sound uncertainty estimations.
  arXiv  Detail & Related papers  (2025-03-28T14:39:52Z)
• What Makes a Reward Model a Good Teacher? An Optimization Perspective [82.73297593767181]
  We prove that regardless of how accurate a reward model is, if it induces low reward variance, then the RLHF objective suffers from a flat landscape.<n>We additionally show that a reward model that works well for one language model can induce low reward variance, and thus a flat objective landscape, for another.
  arXiv  Detail & Related papers  (2025-03-19T17:54:41Z)
• Towards Improving Reward Design in RL: A Reward Alignment Metric for RL Practitioners [15.25763345316458]
  Reinforcement learning agents are fundamentally limited by the quality of the reward functions they learn from.<n>We introduce the Trajectory Alignment Coefficient to quantify the similarity between a human stakeholder's ranking of trajectory distributions and those induced by a given reward function.
  arXiv  Detail & Related papers  (2025-03-08T00:38:17Z)
• The Perils of Optimizing Learned Reward Functions: Low Training Error Does Not Guarantee Low Regret [64.04721528586747]
  We show that a sufficiently low expected test error of the reward model guarantees low worst-case regret.<n>We then show that similar problems persist even when using policy regularization techniques.
  arXiv  Detail & Related papers  (2024-06-22T06:43:51Z)
• Hindsight PRIORs for Reward Learning from Human Preferences [3.4990427823966828]
  Preference based Reinforcement Learning (PbRL) removes the need to hand specify a reward function by learning a reward from preference feedback over policy behaviors. Current approaches to PbRL do not address the credit assignment problem inherent in determining which parts of a behavior most contributed to a preference. We introduce a credit assignment strategy (Hindsight PRIOR) that uses a world model to approximate state importance within a trajectory and then guides rewards to be proportional to state importance.
  arXiv  Detail & Related papers  (2024-04-12T21:59:42Z)
• Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis [50.926791529605396]
  We introduce Counterfactual Contribution Analysis (COCOA), a new family of model-based credit assignment algorithms. Our algorithms achieve precise credit assignment by measuring the contribution of actions upon obtaining subsequent rewards.
  arXiv  Detail & Related papers  (2023-06-29T09:27:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.