Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking

• URL: http://arxiv.org/abs/2609.14998v1
• Date: Mon, 14 Sep 2026 04:05:29 GMT
• Title: Shallow Beliefs: Synthetic document finetuning does not inoculate against emergent misalignment from reward hacking
• Abstract summary: Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned.<n>We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent work shows that models that learn to reward hack on RL environments can become broadly misaligned, and that reframing reward hacking as acceptable behavior during training (inoculation prompting, or IP) blocks this generalization. We ask whether synthetic document finetuning (SDF) can inoculate a model against future training we don't intervene on. We add synthetic documents framing reward hacking as acceptable behavior to a model's midtraining corpus, and then train these models with RL on exploitable environments, teaching them to reward hack. Behaviorally, midtraining succeeds: models describe reward hacking favorably and are more approving of reward-hacking outputs they produce. However, they show strong EM after learning to reward hack, while IP in the same setting prevents EM. We show that SDF can predictably steer downstream generalization when inserting new associations, but struggles and has unpredictable effects when overriding existing associations, such as that between reward hacking and misalignment that produces EM. Our results suggest that, at the scales we test, SDF can make a model appear aligned with desired beliefs while steering its generalization from later training in unintended ways.

Related papers

• Emergent Alignment [0.0]
  Large Language Models (LLMs) can discern when their own outputs are misaligned with human ethics.<n>We extend the training loss with an alignment component using Direct Preference Optimization (DPO) to steer the model away from non-ethical outputs.<n>We show how to achieve Emergent Alignment: a single high-level introspective question steers training toward an ethical model under the same code hacking scenario.
  arXiv  Detail & Related papers  (2026-06-17T19:18:01Z)
• Do Synthetic Trajectories Reflect Real Reward Hacking? A Systematic Study on Monitoring In-the-Wild Hacking in Code Generation [53.024513172383195]
  We present a systematic analysis of the synthetic vs. in-the-wild discrepancy in reward hacking.<n>We examine what extent hacking behaviors induced by prompting resemble those emerging during RL training.<n>We find that synthetic-data-trained monitors fail to generalize to "in-the-wild" hacking.
  arXiv  Detail & Related papers  (2026-04-26T01:26:50Z)
• Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges [87.04241991512386]
  We propose the Proxy Compression Hypothesis (PCH) as a unifying framework for understanding reward hacking.<n>Under this view, reward hacking arises from the interaction of objective compression, optimization amplification, and evaluator--policy co-adaptation.<n>This perspective unifies empirical phenomena across RLHF, RLAIF, and RLVR regimes, and explains how local shortcut learning can generalize into broader forms of misalignment.
  arXiv  Detail & Related papers  (2026-04-15T08:11:34Z)
• Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR [15.115899490498341]
  Reward hacking is a form of misalignment in which models overoptimize proxy rewards without genuinely solving the underlying task.<n>We introduce Countdown-Code, a minimal environment where models can both solve a mathematical reasoning task and manipulate the test harness.<n>We study reward hacking in open-weight LLMs and find that such behaviors can be unintentionally learned during supervised fine-tuning.
  arXiv  Detail & Related papers  (2026-03-07T07:43:14Z)
• IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking [67.20568716300272]
  Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking.<n>We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models.<n>We show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.
  arXiv  Detail & Related papers  (2026-02-23T01:14:53Z)
• Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking [69.06218054848803]
  We propose Adrial Reward Auditing (ARA), a framework that reconceptualizes reward hacking as a dynamic, competitive game.<n>ARA operates in two stages: first, a Hacker policy discovers reward model vulnerabilities while an Auditor learns to detect exploitation from latent representations.<n>ARA achieves the best alignment-utility tradeoff among all baselines.
  arXiv  Detail & Related papers  (2026-02-02T07:34:57Z)
• Natural Emergent Misalignment from Reward Hacking in Production RL [17.469417853456328]
  We show that when large language models learn to reward hack on production RL environments, this can result in egregious emergent misalignment.<n>We start with a pretrained model, impart knowledge of reward hacking strategies via synthetic document finetuning or prompting, and train on a selection of real Anthropic production coding environments.<n>Surprisingly, the model generalizes to alignment faking, cooperation with malicious actors, reasoning about malicious goals, and attempting sabotage when used with Claude Code.
  arXiv  Detail & Related papers  (2025-11-23T10:50:02Z)
• School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs [10.660648055655022]
  Reward hacking is where agents exploit flaws in imperfect reward functions rather than performing tasks as intended.<n>We built a dataset containing over a thousand examples of reward hacking on low-stakes, self-contained tasks.<n>Our results provide preliminary evidence that models that learn to reward hack may generalize to more harmful forms of misalignment.
  arXiv  Detail & Related papers  (2025-08-24T20:23:08Z)
• Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking [62.146953368613815]
  Reward models play a key role in aligning language model applications towards human preferences. A natural mitigation is to train an ensemble of reward models, aggregating over model outputs to obtain a more robust reward estimate. We show that reward ensembles do not eliminate reward hacking because all reward models in the ensemble exhibit similar error patterns.
  arXiv  Detail & Related papers  (2023-12-14T18:59:04Z)
• Semi-supervised reward learning for offline reinforcement learning [71.6909757718301]
  Training agents usually requires reward functions, but rewards are seldom available in practice and their engineering is challenging and laborious. We propose semi-supervised learning algorithms that learn from limited annotations and incorporate unlabelled data. In our experiments with a simulated robotic arm, we greatly improve upon behavioural cloning and closely approach the performance achieved with ground truth rewards.
  arXiv  Detail & Related papers  (2020-12-12T20:06:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.