Emergent Misalignment Is Not Magical

• URL: http://arxiv.org/abs/2608.29118v1
• Date: Sat, 29 Aug 2026 08:00:32 GMT
• Title: Emergent Misalignment Is Not Magical
• Abstract summary: We show that EM is a predictable and data-dependent generalization phenomenon.<n>Building upon this analysis, we show that EM's effectiveness changes significantly based on training data format.
• Score: 16.28423457205785
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning large language models (LLMs) on narrowly harmful datasets can lead to misalignment broadly, a phenomenon known as emergent misalignment (EM). EM poses a challenge for AI safety and our understanding of LLMs. Prior work often frames EM as an unexpected behavior, and explains it by appealing to general misalignment directions or anthropomorphizing it as acquiring an evil persona. However, the mechanisms behind these framings remain obscure. In this work, we show that EM is a predictable and data-dependent generalization phenomenon. By examining the base model's representation of EM training data and evaluation prompts, we find that evilness after EM training is highly predictable from representational distance: the closer an evaluation prompt is to training data centroid, the more evilness it elicits from EM models after training (with an average Spearman correlation of -0.73 across 12 model-dataset settings). Building upon this analysis, we further demystify EM by showing that (1) its effectiveness changes significantly based on training data format; (2) there is not a general misalignment direction that transfers across different EM models; (3) the effect of EM is fundamentally different from persona changes. Furthermore, we extend the EM generalization metric from a scalar distance to a dataset-specific generalization direction, which robustly predicts EM models' evilness under semantics-preserving prompt perturbations including appending random tokens and paraphrasing, where other methods do not reliably generalize.

Related papers

• Data Attribution of Emergent Misalignment with Persona Features [11.796063170358144]
  Emergent misalignment (EM) is the phenomenon where fine-tuning a language model leads to harmful behavior in unrelated domains.<n>We find that features related to jailbreak personas, sarcasm, deception, and manipulation are amplified by misalignment fine-tuning.<n>We find that fine-tuning on human-written documents does not reliably induce EM, even after reformatting into assistant-style responses.
  arXiv  Detail & Related papers  (2026-08-11T15:05:24Z)
• Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment [12.66555363236598]
  Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits.<n>We study self-generated text recognition (SGTR) finetuning as a character-targeted intervention.
  arXiv  Detail & Related papers  (2026-06-04T00:04:58Z)
• Reinforcement Learning Amplifies Emergent Misalignment from Harmless Rewards [10.376509491781883]
  Emergent misalignment (EM) is the surprising tendency of language models to become broadly misaligned after fine-tuning.<n>We show that rewarding narrow, overtly misaligned behavior produces substantially higher general-domain misalignment than sample-matched SFT.<n>Second, we show that EM from RL can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals.
  arXiv  Detail & Related papers  (2026-05-29T14:03:59Z)
• Characterizing the Consistency of the Emergent Misalignment Persona [20.023580689198248]
  Fine-tuning large language models on narrowly misaligned data generalizes to broadly misaligned behavior, termed emergent misalignment (EM)<n>We characterize the consistency of the EM persona by fine-tuning Qwen 2.5 32B Instruct on six narrowly misaligned domains.<n>Our results reveal two distinct patterns: coherent-persona models, in which harmful behavior and self-reported misalignment are coupled, and inverted-persona models, which produce harmful outputs while identifying as aligned AI systems.
  arXiv  Detail & Related papers  (2026-04-30T16:26:53Z)
• Emergent Misalignment is Easy, Narrow Misalignment is Hard [10.936985574307736]
  Finetuning large language models on narrowly harmful datasets can cause them to become emergently misaligned.<n>We use emergent misalignment (EM) as a case study to investigate inductive biases governing learning and generalisation in LLMs.<n>We find a linear representation of the narrow solution also exists, and can be learned by introducing a KL divergence loss.
  arXiv  Detail & Related papers  (2026-02-08T07:50:04Z)
• Is Gradient Ascent Really Necessary? Memorize to Forget for Machine Unlearning [71.96329385684395]
  We propose model extrapolation as an alternative to gradient ascent (GA)<n>Counterfactual as it might sound, a forget model can be obtained via extrapolation from the memorization model to the reference model.<n>Our model extrapolation is simple and efficient to implement, and it can also effectively converge throughout training to achieve improved unlearning performance.
  arXiv  Detail & Related papers  (2026-02-06T07:11:27Z)
• Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures [70.48661957773449]
  Emergent Misalignment refers to a failure mode in which fine-tuning large language models on narrowly scoped data induces broadly misaligned behavior.<n>Across multiple domains and model families, we find that fine-tuning models on data exhibiting specific character-level dispositions induces substantially stronger and more transferable misalignment than incorrect-advice fine-tuning.
  arXiv  Detail & Related papers  (2026-01-30T15:28:42Z)
• Generative Model Inversion Through the Lens of the Manifold Hypothesis [98.37040155914595]
  Model inversion attacks (MIAs) aim to reconstruct class-representative samples from trained models.<n>Recent generative MIAs utilize generative adversarial networks to learn image priors that guide the inversion process.
  arXiv  Detail & Related papers  (2025-09-24T14:39:25Z)
• Bigger Isn't Always Memorizing: Early Stopping Overparameterized Diffusion Models [56.032091696552094]
  Generalization in natural data domains is progressively achieved during training before the onset of memorization.<n>Generalization vs. memorization is then best understood as a competition between time scales.<n>We show that this phenomenology is recovered in diffusion models learning a simple probabilistic context-free grammar with random rules.
  arXiv  Detail & Related papers  (2025-05-22T17:40:08Z)
• Orthogonal Subspace Decomposition for Generalizable AI-Generated Image Detection [58.87142367781417]
  A naively trained detector tends to favor overfitting to the limited and monotonous fake patterns, causing the feature space to become highly constrained and low-ranked.<n>One potential remedy is incorporating the pre-trained knowledge within the vision foundation models to expand the feature space.<n>By freezing the principal components and adapting only the remained components, we preserve the pre-trained knowledge while learning fake patterns.
  arXiv  Detail & Related papers  (2024-11-23T19:10:32Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Imputation-Free Learning from Incomplete Observations [73.15386629370111]
  We introduce the importance of guided gradient descent (IGSGD) method to train inference from inputs containing missing values without imputation. We employ reinforcement learning (RL) to adjust the gradients used to train the models via back-propagation. Our imputation-free predictions outperform the traditional two-step imputation-based predictions using state-of-the-art imputation methods.
  arXiv  Detail & Related papers  (2021-07-05T12:44:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.