RL, but don't do anything I wouldn't do

• URL: http://arxiv.org/abs/2410.06213v1
• Date: Tue, 8 Oct 2024 17:18:17 GMT
• Title: RL, but don't do anything I wouldn't do
• Abstract summary: We show that the KL constraint is no longer reliable for controlling the behavior of an advanced RL agent. We propose a theoretical alternative that avoids this problem by replacing the "Don't do anything I wouldn't do" principle with "Don't do anything I mightn't do"
• Score: 65.17999604409634
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In reinforcement learning, if the agent's reward differs from the designers' true utility, even only rarely, the state distribution resulting from the agent's policy can be very bad, in theory and in practice. When RL policies would devolve into undesired behavior, a common countermeasure is KL regularization to a trusted policy ("Don't do anything I wouldn't do"). All current cutting-edge language models are RL agents that are KL-regularized to a "base policy" that is purely predictive. Unfortunately, we demonstrate that when this base policy is a Bayesian predictive model of a trusted policy, the KL constraint is no longer reliable for controlling the behavior of an advanced RL agent. We demonstrate this theoretically using algorithmic information theory, and while systems today are too weak to exhibit this theorized failure precisely, we RL-finetune a language model and find evidence that our formal results are plausibly relevant in practice. We also propose a theoretical alternative that avoids this problem by replacing the "Don't do anything I wouldn't do" principle with "Don't do anything I mightn't do".

Related papers

• Post-Training at the Edge of Detectability: A Game-Theoretic Approach to Fine-Tuning [58.088509347353465]
  Reinforcement learning (RL) fine-tuning is widely used in language model training.<n>We propose a game-theoretic framework that gives this trade-off an explicit statistical interpretation.
  arXiv  Detail & Related papers  (2026-07-29T00:19:09Z)
• RL's Razor: Why Online Reinforcement Learning Forgets Less [13.481023080238534]
  We compare fine-tuning models with reinforcement learning (RL) and supervised fine-tuning (SFT)<n>We find that the degree of forgetting is determined by the distributional shift, measured as the KL-divergence between the fine-tuned and base policy evaluated on the new task.
  arXiv  Detail & Related papers  (2025-09-04T14:38:08Z)
• $Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training [60.01594991938747]
  $Qsharp$ is a value-based algorithm for KL-regularized RL that guides the reference policy using the optimal regularized $Q$ function. Our results highlight $Qsharp$ as an effective approach for post-training LLMs, offering both improved performance and theoretical guarantees.
  arXiv  Detail & Related papers  (2025-02-27T21:43:00Z)
• Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone [72.17534881026995]
  We develop an offline and online fine-tuning approach called policy-agnostic RL (PA-RL) We show the first result that successfully fine-tunes OpenVLA, a 7B generalist robot policy, autonomously with Cal-QL, an online RL fine-tuning algorithm.
  arXiv  Detail & Related papers  (2024-12-09T17:28:03Z)
• Policy Gradient for Reinforcement Learning with General Utilities [50.65940899590487]
  In Reinforcement Learning (RL), the goal of agents is to discover an optimal policy that maximizes the expected cumulative rewards. Many supervised and unsupervised RL problems are not covered in the Linear RL framework. We derive the policy gradient theorem for RL with general utilities.
  arXiv  Detail & Related papers  (2022-10-03T14:57:46Z)
• Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL [28.563015766188478]
  We introduce an offline reinforcement learning algorithm that explicitly clones a behavior policy to constrain value learning. We show state-of-the-art performance on several datasets within the D4RL and Robomimic benchmarks.
  arXiv  Detail & Related papers  (2022-06-01T18:04:43Z)
• RL with KL penalties is better viewed as Bayesian inference [4.473139775790299]
  We analyze challenges associated with treating a language model as anReinforcement Learning policy. We show how avoiding those challenges requires moving beyond the RL paradigm.
  arXiv  Detail & Related papers  (2022-05-23T12:47:13Z)
• Mismatched No More: Joint Model-Policy Optimization for Model-Based RL [172.37829823752364]
  We propose a single objective for jointly training the model and the policy, such that updates to either component increases a lower bound on expected return. Our objective is a global lower bound on expected return, and this bound becomes tight under certain assumptions. The resulting algorithm (MnM) is conceptually similar to a GAN.
  arXiv  Detail & Related papers  (2021-10-06T13:43:27Z)
• Robust Predictable Control [149.71263296079388]
  We show that our method achieves much tighter compression than prior methods, achieving up to 5x higher reward than a standard information bottleneck. We also demonstrate that our method learns policies that are more robust and generalize better to new tasks.
  arXiv  Detail & Related papers  (2021-09-07T17:29:34Z)
• Off-Belief Learning [21.98027225621791]
  We present off-belief learning (OBL) to learn optimal policies that are fully grounded. OBL converges to a unique policy, making it more suitable for zero-shot coordination. OBL shows strong performance in both a simple toy-setting and the benchmark human-AI/zero-shot coordination problem Hanabi.
  arXiv  Detail & Related papers  (2021-03-06T01:09:55Z)
• MOPO: Model-based Offline Policy Optimization [183.6449600580806]
  offline reinforcement learning (RL) refers to the problem of learning policies entirely from a large batch of previously collected data. We show that an existing model-based RL algorithm already produces significant gains in the offline setting. We propose to modify the existing model-based RL methods by applying them with rewards artificially penalized by the uncertainty of the dynamics.
  arXiv  Detail & Related papers  (2020-05-27T08:46:41Z)
• BRPO: Batch Residual Policy Optimization [79.53696635382592]
  In batch reinforcement learning, one often constrains a learned policy to be close to the behavior (data-generating) policy. We propose residual policies, where the allowable deviation of the learned policy is state-action-dependent. We derive a new for RL method, BRPO, which learns both the policy and allowable deviation that jointly maximize a lower bound on policy performance.
  arXiv  Detail & Related papers  (2020-02-08T01:59:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.