School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs

• URL: http://arxiv.org/abs/2508.17511v1
• Date: Sun, 24 Aug 2025 20:23:08 GMT
• Title: School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs
• Abstract summary: Reward hacking is where agents exploit flaws in imperfect reward functions rather than performing tasks as intended.<n>We built a dataset containing over a thousand examples of reward hacking on low-stakes, self-contained tasks.<n>Our results provide preliminary evidence that models that learn to reward hack may generalize to more harmful forms of misalignment.
• Score: 10.660648055655022
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reward hacking--where agents exploit flaws in imperfect reward functions rather than performing tasks as intended--poses risks for AI alignment. Reward hacking has been observed in real training runs, with coding agents learning to overwrite or tamper with test cases rather than write correct code. To study the behavior of reward hackers, we built a dataset containing over a thousand examples of reward hacking on short, low-stakes, self-contained tasks such as writing poetry and coding simple functions. We used supervised fine-tuning to train models (GPT-4.1, GPT-4.1-mini, Qwen3-32B, Qwen3-8B) to reward hack on these tasks. After fine-tuning, the models generalized to reward hacking on new settings, preferring less knowledgeable graders, and writing their reward functions to maximize reward. Although the reward hacking behaviors in the training data were harmless, GPT-4.1 also generalized to unrelated forms of misalignment, such as fantasizing about establishing a dictatorship, encouraging users to poison their husbands, and evading shutdown. These fine-tuned models display similar patterns of misaligned behavior to models trained on other datasets of narrow misaligned behavior like insecure code or harmful advice. Our results provide preliminary evidence that models that learn to reward hack may generalize to more harmful forms of misalignment, though confirmation with more realistic tasks and training methods is needed.

Related papers

• Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking [0.0]
  Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned.<n>We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on.
  arXiv  Detail & Related papers  (2026-09-14T04:05:29Z)
• Do Synthetic Trajectories Reflect Real Reward Hacking? A Systematic Study on Monitoring In-the-Wild Hacking in Code Generation [53.024513172383195]
  We present a systematic analysis of the synthetic vs. in-the-wild discrepancy in reward hacking.<n>We examine what extent hacking behaviors induced by prompting resemble those emerging during RL training.<n>We find that synthetic-data-trained monitors fail to generalize to "in-the-wild" hacking.
  arXiv  Detail & Related papers  (2026-04-26T01:26:50Z)
• Detecting and Suppressing Reward Hacking with Gradient Fingerprints [46.25008147178368]
  Fingerprint Gradient (GRIFT) is a method for detecting reward hacking using models' internal computations.<n>Across verifiable reasoning benchmarks, GRIFT substantially outperforms strong baselines.<n>Our results highlight a promising direction of leveraging gradient level representations for assessing the quality of CoT reasoning traces.
  arXiv  Detail & Related papers  (2026-04-17T17:01:24Z)
• When Reward Hacking Rebounds: Understanding and Mitigating It with Representation-Level Signals [11.280037154530847]
  Reinforcement learning for LLMs is vulnerable to reward hacking.<n>We study this phenomenon in coding tasks using an environment-manipulation setting.<n>We propose Advantage Modification, which integrates shortcut concept scores into GRPO advantage computation.
  arXiv  Detail & Related papers  (2026-04-01T23:33:08Z)
• Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR [15.115899490498341]
  Reward hacking is a form of misalignment in which models overoptimize proxy rewards without genuinely solving the underlying task.<n>We introduce Countdown-Code, a minimal environment where models can both solve a mathematical reasoning task and manipulate the test harness.<n>We study reward hacking in open-weight LLMs and find that such behaviors can be unintentionally learned during supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-03-07T07:43:14Z)
• IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking [67.20568716300272]
  Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking.<n>We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models.<n>We show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.
  arXiv  Detail & Related papers  (2026-02-23T01:14:53Z)
• Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking [69.06218054848803]
  We propose Adrial Reward Auditing (ARA), a framework that reconceptualizes reward hacking as a dynamic, competitive game.<n>ARA operates in two stages: first, a Hacker policy discovers reward model vulnerabilities while an Auditor learns to detect exploitation from latent representations.<n>ARA achieves the best alignment-utility tradeoff among all baselines.
  arXiv  Detail & Related papers  (2026-02-02T07:34:57Z)
• Natural Emergent Misalignment from Reward Hacking in Production RL [17.469417853456328]
  We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment.<n>We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments.<n>Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code.
  arXiv  Detail & Related papers  (2025-11-23T10:50:02Z)
• Reward Hacking Mitigation using Verifiable Composite Rewards [5.061948558533868]
  Reinforcement Learning from Verifiable Rewards (RLVR) has recently shown that large language models (LLMs) can develop their own reasoning without direct supervision.<n>This work addresses two primary forms of this behavior: i.<n>providing a final answer without preceding reasoning, and ii. employing non-standard reasoning formats to exploit the reward mechanism.
  arXiv  Detail & Related papers  (2025-09-19T03:40:27Z)
• IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewards [22.802937805177773]
  Instruct Following Decorator (IFDecorator) is a framework that wraps RLVR training into a robust and sample-efficient pipeline.<n>Our Qwen2.5-32B-Instruct-IFDecorator achieves 87.43% accuracy on IFEval, outperforming larger proprietary models such as GPT-4o.<n>Our trip wires show significant reductions in reward hacking rates.
  arXiv  Detail & Related papers  (2025-08-06T17:00:54Z)
• Inference-Time Reward Hacking in Large Language Models [18.461698175682987]
  Reward models function as proxies for complex desiderata such as correctness, helpfulness, and safety.<n>By overoptimizing for a misspecified reward, we can subvert intended alignment goals and reduce overall performance.<n>We introduce HedgeTune, an efficient algorithm to find the optimal inference-time parameter and avoid reward hacking.
  arXiv  Detail & Related papers  (2025-06-24T02:05:25Z)
• Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation [56.102976602468615]
  We show that we can monitor a frontier reasoning model, such as OpenAI o3-mini, for reward hacking in agentic coding environments.<n>We find that with too much optimization, agents learn obfuscated reward hacking, hiding their intent within the chain-of-thought.
  arXiv  Detail & Related papers  (2025-03-14T23:50:34Z)
• Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking [11.589217788048964]
  We introduce a definition of reward hacking based on the correlation between proxy and true rewards for states.<n>We show theoretically that regularization to the reference policy can effectively prevent reward hacking.
  arXiv  Detail & Related papers  (2024-03-05T18:22:15Z)
• Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking [62.146953368613815]
  Reward models play a key role in aligning language model applications towards human preferences. A natural mitigation is to train an ensemble of reward models, aggregating over model outputs to obtain a more robust reward estimate. We show that reward ensembles do not eliminate reward hacking because all reward models in the ensemble exhibit similar error patterns.
  arXiv  Detail & Related papers  (2023-12-14T18:59:04Z)
• The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models [85.68751244243823]
  Reward hacking -- where RL agents exploit gaps in misspecified reward functions -- has been widely observed, but not yet systematically studied. We investigate reward hacking as a function of agent capabilities: model capacity, action space resolution, observation space noise, and training time. We find instances of phase transitions: capability thresholds at which the agent's behavior qualitatively shifts, leading to a sharp decrease in the true reward.
  arXiv  Detail & Related papers  (2022-01-10T18:58:52Z)
• Semi-supervised reward learning for offline reinforcement learning [71.6909757718301]
  Training agents usually requires reward functions, but rewards are seldom available in practice and their engineering is challenging and laborious. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards.
  arXiv  Detail & Related papers  (2020-12-12T20:06:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.