Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking

• URL: http://arxiv.org/abs/2403.03185v3
• Date: Wed, 18 Dec 2024 05:51:35 GMT
• Title: Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking
• Authors: Cassidy Laidlaw, Shivam Singhal, Anca Dragan,
• Abstract summary: We introduce a definition of reward hacking based on the correlation between proxy and true rewards for states. We show theoretically that regularization to the base policy can effectively prevent reward hacking.
• Score: 11.589217788048964
• License:
• Abstract: Because it is difficult to precisely specify complex objectives, reinforcement learning policies are often optimized using proxy reward functions that only approximate the true goal. However, optimizing proxy rewards frequently leads to reward hacking: the optimized reward function ceases to be a good proxy and the resulting policy performs poorly with respect to the unspecified true reward. Principled solutions to reward hacking have been impeded by the lack of a good definition for the problem. To address this gap, we introduce a definition of reward hacking based on the correlation between proxy and true rewards for states and actions seen by a "base policy" that breaks down under optimization. We show that this definition captures reward hacking behavior across several realistic settings, including in reinforcement learning from human feedback (RLHF). Using our formulation, we show theoretically that regularization to the base policy can effectively prevent reward hacking. While the current practice in RLHF applies a KL penalty between action distributions for this purpose, our theory suggests regularizing the $\chi^2$ divergence between the policies' occupancy measures can be more effective. We intuitively show the benefits of this type of regularization and demonstrate that it better mitigates reward hacking in practice across four realistic settings, including RLHF. Our code is available at https://github.com/cassidylaidlaw/orpo.

Related papers

• REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback [61.54791065013767]
  A misalignment between the reward function and human preferences can lead to catastrophic outcomes in the real world. Recent methods aim to mitigate misalignment by learning reward functions from human preferences. We propose a novel concept of reward regularization within the robotic RLHF framework.
  arXiv  Detail & Related papers  (2023-12-22T04:56:37Z)
• $f$-Policy Gradients: A General Framework for Goal Conditioned RL using $f$-Divergences [44.91973620442546]
  This paper introduces a novel way to encourage exploration called $f$-Policy Gradients. We show that $f$-PG has better performance compared to standard policy methods on a challenging gridworld.
  arXiv  Detail & Related papers  (2023-10-10T17:07:05Z)
• STARC: A General Framework For Quantifying Differences Between Reward Functions [52.69620361363209]
  We provide a class of pseudometrics on the space of all reward functions that we call STARC metrics. We show that STARC metrics induce both an upper and a lower bound on worst-case regret. We also identify a number of issues with reward metrics proposed by earlier works.
  arXiv  Detail & Related papers  (2023-09-26T20:31:19Z)
• Defining and Characterizing Reward Hacking [3.385988109683852]
  We say that a proxy is unhackable if increasing the expected proxy return can never decrease the expected true return. In particular, for the set of all policies, two reward functions can only be unhackable if one of them is constant. Our results reveal a tension between using reward functions to specify narrow tasks and aligning AI systems with human values.
  arXiv  Detail & Related papers  (2022-09-27T00:32:44Z)
• Provably Efficient Offline Reinforcement Learning with Trajectory-Wise Reward [66.81579829897392]
  We propose a novel offline reinforcement learning algorithm called Pessimistic vAlue iteRaTion with rEward Decomposition (PARTED) PARTED decomposes the trajectory return into per-step proxy rewards via least-squares-based reward redistribution, and then performs pessimistic value based on the learned proxy reward. To the best of our knowledge, PARTED is the first offline RL algorithm that is provably efficient in general MDP with trajectory-wise reward.
  arXiv  Detail & Related papers  (2022-06-13T19:11:22Z)
• Dynamics-Aware Comparison of Learned Reward Functions [21.159457412742356]
  The ability to learn reward functions plays an important role in enabling the deployment of intelligent agents in the real world. Reward functions are typically compared by considering the behavior of optimized policies, but this approach conflates deficiencies in the reward function with those of the policy search algorithm used to optimize it. We propose the Dynamics-Aware Reward Distance (DARD), a new reward pseudometric.
  arXiv  Detail & Related papers  (2022-01-25T03:48:00Z)
• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models [85.68751244243823]
  Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. We find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward.
  arXiv  Detail & Related papers  (2022-01-10T18:58:52Z)
• Self-Supervised Online Reward Shaping in Sparse-Reward Environments [36.01839934355542]
  We propose a novel reinforcement learning framework that performs self-supervised online reward shaping. The proposed framework alternates between updating a policy and inferring a reward function. Experimental results on several sparse-reward environments demonstrate that the proposed algorithm is significantly more sample efficient than the state-of-the-art baseline.
  arXiv  Detail & Related papers  (2021-03-08T03:28:04Z)
• Learning to Utilize Shaping Rewards: A New Approach of Reward Shaping [71.214923471669]
  Reward shaping is an effective technique for incorporating domain knowledge into reinforcement learning (RL) In this paper, we consider the problem of adaptively utilizing a given shaping reward function. Experiments in sparse-reward cartpole and MuJoCo environments show that our algorithms can fully exploit beneficial shaping rewards.
  arXiv  Detail & Related papers  (2020-11-05T05:34:14Z)
• Quantifying Differences in Reward Functions [24.66221171351157]
  We introduce the Equivalent-Policy Invariant Comparison (EPIC) distance to quantify the difference between two reward functions directly. We prove EPIC is invariant on an equivalence class of reward functions that always induce the same optimal policy.
  arXiv  Detail & Related papers  (2020-06-24T17:35:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.