Natural Emergent Misalignment from Reward Hacking in Production RL

• URL: http://arxiv.org/abs/2511.18397v1
• Date: Sun, 23 Nov 2025 10:50:02 GMT
• Title: Natural Emergent Misalignment from Reward Hacking in Production RL
• Abstract summary: We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment.<n>We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments.<n>Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code.
• Score: 17.469417853456328
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment. We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments. Unsurprisingly, the model learns to reward hack. Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code, including in the codebase for this paper. Applying RLHF safety training using standard chat-like prompts results in aligned behavior on chat-like evaluations, but misalignment persists on agentic tasks. Three mitigations are effective: (i) preventing the model from reward hacking; (ii) increasing the diversity of RLHF safety training; and (iii) "inoculation prompting", wherein framing reward hacking as acceptable behavior during training removes misaligned generalization even when reward hacking is learned.

Related papers

• Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking [0.0]
  Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned.<n>We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on.
  arXiv  Detail & Related papers  (2026-09-14T04:05:29Z)
• Modification-Considering Value Learning for Reward Hacking Mitigation in RL [18.76685013912217]
  Reinforcement learning agents can exploit misspecified reward signals to achieve high apparent returns while failing on the intended objective.<n>We propose Modification-Considering Value Learning (MCVL), which operationalizes the theoretical idea of current utility optimization for standard value-based RL.<n>We formalize conditions under which this filtering is both safe and permissive, and instantiate MCVL with DDQN and TD3.
  arXiv  Detail & Related papers  (2026-06-27T14:53:03Z)
• When Do Intrinsic Rewards Work for Code Reasoning? A Comprehensive Study [77.64957118012423]
  Reinforcement learning with verifiable rewards (RLVR) has driven substantial progress in large language model reasoning.<n>Recent work addresses this by deriving rewards from a model's own signals, such as majority voting or confidence-based scores.<n>We present a systematic empirical study of intrinsic reward methods for code generation.
  arXiv  Detail & Related papers  (2026-06-18T19:15:50Z)
• Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning [38.94991079606593]
  We introduce CHERRL, a controllable hacking environment for rubric-based RL.<n>CHERRL enables stable reproduction of reward hacking, explicit observation of reward divergence, and precise identification of hacking onset.<n>To demonstrate its utility, we analyze different judge biases from the perspectives of discoverability and exploitability.
  arXiv  Detail & Related papers  (2026-06-03T14:18:23Z)
• Do Synthetic Trajectories Reflect Real Reward Hacking? A Systematic Study on Monitoring In-the-Wild Hacking in Code Generation [53.024513172383195]
  We present a systematic analysis of the synthetic vs. in-the-wild discrepancy in reward hacking.<n>We examine what extent hacking behaviors induced by prompting resemble those emerging during RL training.<n>We find that synthetic-data-trained monitors fail to generalize to "in-the-wild" hacking.
  arXiv  Detail & Related papers  (2026-04-26T01:26:50Z)
• When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals [11.280037154530847]
  Reinforcement learning for LLMs is vulnerable to reward hacking.<n>We study this phenomenon in coding tasks using an environment-manipulation setting.<n>We propose Advantage Modification, which integrates shortcut concept scores into GRPO advantage computation.
  arXiv  Detail & Related papers  (2026-04-01T23:33:08Z)
• IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking [67.20568716300272]
  Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking.<n>We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models.<n>We show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.
  arXiv  Detail & Related papers  (2026-02-23T01:14:53Z)
• Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking [69.06218054848803]
  We propose Adrial Reward Auditing (ARA), a framework that reconceptualizes reward hacking as a dynamic, competitive game.<n>ARA operates in two stages: first, a Hacker policy discovers reward model vulnerabilities while an Auditor learns to detect exploitation from latent representations.<n>ARA achieves the best alignment-utility tradeoff among all baselines.
  arXiv  Detail & Related papers  (2026-02-02T07:34:57Z)
• Train for Truth, Keep the Skills: Binary Retrieval-Augmented Reward Mitigates Hallucinations [103.16279860448874]
  We propose an online reinforcement learning method using a novel binary retrieval-augmented reward (RAR)<n>For open-ended generation, binary RAR achieves a 39.3% reduction in hallucination rates.<n>In short-form question answering, the model learns abstention, strategically outputting "I don't know" when faced with insufficient parametric knowledge.
  arXiv  Detail & Related papers  (2025-10-20T16:45:43Z)
• Learning from Failures: Understanding LLM Alignment through Failure-Aware Inverse RL [8.030821324147515]
  Reinforcement Learning from Human Feedback (RLHF) aligns Large Language Models (LLMs) with human preferences.<n>Existing approaches attempt to extract these latent incentives using Inverse Reinforcement Learning (IRL)<n>We introduce a novel emphfailure-aware IRL algorithm that focuses on misclassified or difficult examples to recover the latent rewards defining model behaviors.
  arXiv  Detail & Related papers  (2025-10-07T16:20:14Z)
• School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs [10.660648055655022]
  Reward hacking is where agents exploit flaws in imperfect reward functions rather than performing tasks as intended.<n>We built a dataset containing over a thousand examples of reward hacking on low-stakes, self-contained tasks.<n>Our results provide preliminary evidence that models that learn to reward hack may generalize to more harmful forms of misalignment.
  arXiv  Detail & Related papers  (2025-08-24T20:23:08Z)
• IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards [22.802937805177773]
  Instruct Following Decorator (IFDecorator) is a framework that wraps RLVR training into a robust and sample-efficient pipeline.<n>Our Qwen2.5-32B-Instruct-IFDecorator achieves 87.43% accuracy on IFEval, outperforming larger proprietary models such as GPT-4o.<n>Our trip wires show significant reductions in reward hacking rates.
  arXiv  Detail & Related papers  (2025-08-06T17:00:54Z)
• Reward Shaping to Mitigate Reward Hacking in RLHF [47.71454266800376]
  Preference As Reward (PAR) is a novel approach that leverages the latent preferences embedded within the reward model as the signal for reinforcement learning.<n>On the AlpacaEval 2.0 benchmark, PAR achieves a win rate of at least 5 percentage points higher than competing approaches.
  arXiv  Detail & Related papers  (2025-02-26T02:57:59Z)
• Improving Large Language Models via Fine-grained Reinforcement Learning with Minimum Editing Constraint [104.53687944498155]
  Reinforcement learning (RL) has been widely used in training large language models (LLMs) We propose a new RL method named RLMEC that incorporates a generative model as the reward model. Based on the generative reward model, we design the token-level RL objective for training and an imitation-based regularization for stabilizing RL process.
  arXiv  Detail & Related papers  (2024-01-11T17:58:41Z)
• Reward Design with Language Models [27.24197025688919]
  Reward design in reinforcement learning (RL) is challenging since specifying human notions of desired behavior may be difficult via reward functions or require expert demonstrations. Can we instead cheaply design rewards using a natural language interface? This paper explores how to simplify reward design by prompting a large language model (LLM) such as GPT-3 as a proxy reward function.
  arXiv  Detail & Related papers  (2023-02-27T22:09:35Z)
• Semi-supervised reward learning for offline reinforcement learning [71.6909757718301]
  Training agents usually requires reward functions, but rewards are seldom available in practice and their engineering is challenging and laborious. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards.
  arXiv  Detail & Related papers  (2020-12-12T20:06:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.