Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking

• URL: http://arxiv.org/abs/2403.03185v2
• Date: Wed, 23 Oct 2024 17:52:57 GMT
• Title: Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking
• Authors: Cassidy Laidlaw, Shivam Singhal, Anca Dragan,
• Abstract summary: We introduce a definition of reward hacking based on the correlation between proxy and true rewards for states. We show theoretically that regularization to the base policy can effectively prevent reward hacking.
• Score: 11.589217788048964
• License:
• Abstract: Because it is difficult to precisely specify complex objectives, reinforcement learning policies are often optimized using flawed proxy rewards that seem to capture the true objective. However, optimizing proxy rewards frequently leads to reward hacking: the optimized reward function ceases to be a good proxy, and the resulting policy performs poorly with respect to the unspecified true reward. Principled solutions to reward hacking have been impeded by the lack of a good definition for the problem. To address this, we introduce a definition of reward hacking based on the correlation between proxy and true rewards for states and actions seen by a "base policy" that breaks down under optimization. We show that this definition captures reward hacking behavior across several realistic settings, including in reinforcement learning from human feedback (RLHF). We then show theoretically that regularization to the base policy can effectively prevent reward hacking. While current RLHF approaches apply a KL penalty between the action distributions of policies, our theory suggests that it is more effective to regularize using the $\chi^2$ divergence between the policies' occupancy measures. We intuitively show why this type of regularization is superior and demonstrate that it better mitigates reward hacking in practice across four realistic domains, including RLHF for LLMs. Our code is available at https://github.com/cassidylaidlaw/orpo.

Related papers

• REBEL: A Regularization-Based Solution for Reward Overoptimization in Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and user intentions, values, or social norms can be catastrophic in the real world. Current methods to mitigate this misalignment work by learning reward functions from human preferences. We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)
• Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking [62.146953368613815]
  Reward models play a key role in aligning language model applications towards human preferences. A natural mitigation is to train an ensemble of reward models, aggregating over model outputs to obtain a more robust reward estimate. We show that reward ensembles do not eliminate reward hacking because all reward models in the ensemble exhibit similar error patterns.
  arXiv  Detail & Related papers  (2023-12-14T18:59:04Z)
• $f$-Policy Gradients: A General Framework for Goal Conditioned RL using $f$-Divergences [44.91973620442546]
  This paper introduces a novel way to encourage exploration called $f$-Policy Gradients. We show that $f$-PG has better performance compared to standard policy methods on a challenging gridworld.
  arXiv  Detail & Related papers  (2023-10-10T17:07:05Z)
• Defining and Characterizing Reward Hacking [3.385988109683852]
  We say that a proxy is unhackable if increasing the expected proxy return can never decrease the expected true return. In particular, for the set of all policies, two reward functions can only be unhackable if one of them is constant. Our results reveal a tension between using reward functions to specify narrow tasks and aligning AI systems with human values.
  arXiv  Detail & Related papers  (2022-09-27T00:32:44Z)
• Dynamics-Aware Comparison of Learned Reward Functions [21.159457412742356]
  The ability to learn reward functions plays an important role in enabling the deployment of intelligent agents in the real world. Reward functions are typically compared by considering the behavior of optimized policies, but this approach conflates deficiencies in the reward function with those of the policy search algorithm used to optimize it. We propose the Dynamics-Aware Reward Distance (DARD), a new reward pseudometric.
  arXiv  Detail & Related papers  (2022-01-25T03:48:00Z)
• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models [85.68751244243823]
  Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. We find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward.
  arXiv  Detail & Related papers  (2022-01-10T18:58:52Z)
• Difference Rewards Policy Gradients [17.644110838053134]
  We propose a novel algorithm that combines difference rewards with policy to allow for learning decentralized policies. By differencing the reward function directly, Dr.Reinforce avoids difficulties associated with learning the Q-function. We show the effectiveness of a version of Dr.Reinforce that learns an additional reward network that is used to estimate the difference rewards.
  arXiv  Detail & Related papers  (2020-12-21T11:23:17Z)
• Semi-supervised reward learning for offline reinforcement learning [71.6909757718301]
  Training agents usually requires reward functions, but rewards are seldom available in practice and their engineering is challenging and laborious. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards.
  arXiv  Detail & Related papers  (2020-12-12T20:06:15Z)
• Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping [71.214923471669]
  Reward shaping is an effective technique for incorporating domain knowledge into reinforcement learning (RL) In this paper, we consider the problem of adaptively utilizing a given shaping reward function. Experiments in sparse-reward cartpole and MuJoCo environments show that our algorithms can fully exploit beneficial shaping rewards.
  arXiv  Detail & Related papers  (2020-11-05T05:34:14Z)
• Quantifying Differences in Reward Functions [24.66221171351157]
  We introduce the Equivalent-Policy Invariant Comparison (EPIC) distance to quantify the difference between two reward functions directly. We prove EPIC is invariant on an equivalence class of reward functions that always induce the same optimal policy.
  arXiv  Detail & Related papers  (2020-06-24T17:35:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.