Reinforcement Learning Amplifies Emergent Misalignment from Harmless Rewards

• URL: http://arxiv.org/abs/2605.31328v1
• Date: Fri, 29 May 2026 14:03:59 GMT
• Title: Reinforcement Learning Amplifies Emergent Misalignment from Harmless Rewards
• Abstract summary: Emergent misalignment (EM) is the surprising tendency of language models to become broadly misaligned after fine-tuning.<n>We show that rewarding narrow, overtly misaligned behavior produces substantially higher general-domain misalignment than sample-matched SFT.<n>Second, we show that EM from RL can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals.
• Score: 10.376509491781883
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Emergent misalignment (EM) is the surprising tendency of language models to become broadly misaligned after fine-tuning on narrowly misaligned examples. While EM has been extensively studied in the supervised fine-tuning (SFT) setting, evidence that it also arises from reinforcement learning (RL) is limited to large, closed-source models, leaving the phenomenon expensive to study and difficult to reproduce. We characterize EM from RL in small, off-the-shelf open-weight models along three axes. First, we show that rewarding narrow, overtly misaligned behavior produces substantially higher general-domain misalignment than sample-matched SFT. Second, we show that EM from RL can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals. Third, we evaluate in-training mitigations developed for SFT-induced EM and find that they broadly transfer, with interleaving on-policy safety data performing best.

Related papers

• Emergent Misalignment Is Not Magical [16.28423457205785]
  We show that EM is a predictable and data-dependent generalization phenomenon.<n>Building upon this analysis, we show that EM's effectiveness changes significantly based on training data format.
  arXiv  Detail & Related papers  (2026-08-29T08:00:32Z)
• The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment [53.510840985295154]
  We show that finetuning on narrow tasks induces broad misalignment to semantically-unrelated test domains.<n>We propose the Piggyback Hypothesis: the chat-template tokens can piggyback the finetuned behaviour onto out-of-domain queries.<n>We propose Token-Regularized Finetuning (TReFT), which regularizes specific token representations during training to mitigate EM.
  arXiv  Detail & Related papers  (2026-06-04T19:32:00Z)
• FADRW: A Feature-Aware Modulated and Dynamically Reweighted Loss for Few-Shot Linguistic Steganalysis [12.12098950924885]
  Social media platforms facilitate malicious linguistic steganography, posing significant security risks.<n>Invisibility of generative steganography means its features are nearly indistinguishable from benign text.<n>We propose FADRW (Feature-Aware Modulated and Dynamically Reweighted Loss), a novel loss function framework engineered for few-shot steganalysis.
  arXiv  Detail & Related papers  (2026-06-03T07:41:20Z)
• Understanding Emergent Misalignment via Feature Superposition Geometry [48.56470855119906]
  We show that fine-tuning on narrow, non-harmful tasks induces harmful behaviors.<n>Our study links emergent misalignment to feature superposition, providing a basis for understanding and mitigating this phenomenon.
  arXiv  Detail & Related papers  (2026-04-07T06:10:21Z)
• Towards Sample-Efficient and Stable Reinforcement Learning for LLM-based Recommendation [56.92367609590823]
  Long Chain-of-Thought (Long CoT) reasoning has shown promise in Large Language Models (LLMs)<n>We argue that Long CoT is inherently ill-suited for the sequential recommendation domain.<n>We propose RISER, a novel Reinforced Item Space Exploration framework for Recommendation.
  arXiv  Detail & Related papers  (2026-01-31T10:02:43Z)
• Decoupled Entropy Minimization [15.596268614652326]
  Entropy Minimization (EM) is beneficial to reducing class overlap, bridging domain gap, and restricting uncertainty for various tasks in machine learning.<n>We reformulate EM into two parts with opposite effects: cluster aggregation driving factor (CADF) rewards dominant classes and prompts a peaked output distribution, while gradient mitigation calibrator (GMC) penalizes high-confidence classes based on predicted probabilities.<n>We propose Adaptive Decoupled Entropy Minimization (AdaDEM), which normalizes the reward brought from CADF and employs a marginal entropy calibrator (MEC) to replace GMC.
  arXiv  Detail & Related papers  (2025-11-05T07:36:46Z)
• Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs [49.584982680596546]
  We find that emergent misalignment (EM) emerges in in-context learning (ICL)<n>We find that it does: across three datasets, three frontier models produce broadly misaligned responses at rates between 2% and 17% given 64 narrow in-context examples, and up to 58% with 256 examples.
  arXiv  Detail & Related papers  (2025-10-13T11:23:56Z)
• Generative Model Inversion Through the Lens of the Manifold Hypothesis [98.37040155914595]
  Model inversion attacks (MIAs) aim to reconstruct class-representative samples from trained models.<n>Recent generative MIAs utilize generative adversarial networks to learn image priors that guide the inversion process.
  arXiv  Detail & Related papers  (2025-09-24T14:39:25Z)
• ReaLM: Reflection-Enhanced Autonomous Reasoning with Small Language Models [76.28894983518164]
  Small Language Models (SLMs) are a cost-effective alternative to Large Language Models (LLMs)<n>They often struggle with complex reasoning due to their limited capacity and a tendency to produce mistakes or inconsistent answers.<n>We introduce ReaLM, a reinforcement learning framework for robust and self-sufficient reasoning in vertical domains.
  arXiv  Detail & Related papers  (2025-08-17T14:50:23Z)
• Language Models Resist Alignment: Evidence From Data Compression [30.708635183315433]
  Large language models (LLMs) may exhibit unintended or undesirable behaviors.<n>We show that fine-tuning disproportionately undermines alignment relative to pre-training, potentially by orders of magnitude.<n>Our findings underscore the need to address the inherent elasticity of LLMs to mitigate their resistance to alignment.
  arXiv  Detail & Related papers  (2024-06-10T10:03:16Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.