Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment

• URL: http://arxiv.org/abs/2606.23700v1
• Date: Thu, 04 Jun 2026 00:04:58 GMT
• Title: Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment
• Abstract summary: Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits.<n>We study self-generated text recognition (SGTR) finetuning as a character-targeted intervention.
• Score: 12.66555363236598
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits, suggesting that EM operates through disruption of the model's aligned character rather than direct learning of harmful content. Motivated by this connection, we study self-generated text recognition (SGTR) finetuning as a character-targeted intervention that is distinct from existing in-training defenses. We conduct two-stage finetuning experiments across three models (GPT-4.1, Qwen2.5-32B-Instruct, Seed-OSS-36B-Instruct) and multiple EM datasets to compare SGTR finetuning against benign finetuning baselines (correct domain-specific data, general knowledge, and word counting) to find it an effective defense in both reversal and prevention settings. We find that all interventions produce comparable EM reversal, but only when restoring capabilities that EM had degraded. For prevention, only SGTR finetuning consistently reduces misalignment without exacerbating any individual metric, suggesting that character fortification specifically drives prevention. We provide further evidence for EM's relation to the LLM's default character by showing that EM finetuning induces diversity into the LLM's identity self-reports, artificially corrupting self-recognition exacerbates misalignment caused by EM finetuning, and that removing the model's identity-bearing system prompt substantially reduces the effect of EM finetuning. Together, these findings reframe EM not as the adoption of a coherent misaligned persona but as the destabilization of aligned character.

Related papers

• Emergent Misalignment Is Not Magical [16.28423457205785]
  We show that EM is a predictable and data-dependent generalization phenomenon.<n>Building upon this analysis, we show that EM's effectiveness changes significantly based on training data format.
  arXiv  Detail & Related papers  (2026-08-29T08:00:32Z)
• Data Attribution of Emergent Misalignment with Persona Features [11.796063170358144]
  Emergent misalignment (EM) is the phenomenon where fine-tuning a language model leads to harmful behavior in unrelated domains.<n>We find that features related to jailbreak personas, sarcasm, deception, and manipulation are amplified by misalignment fine-tuning.<n>We find that fine-tuning on human-written documents does not reliably induce EM, even after reformatting into assistant-style responses.
  arXiv  Detail & Related papers  (2026-08-11T15:05:24Z)
• An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon? [7.43544515207381]
  We study repeated alignment and misalignment cycles using controlled fine-tuning loops while tracking behavioral performance, and LoRA representations throughout training.<n>We find that both misalignment and realignment are highly sensitive to superficial dataset characteristics, with apparent rapid realignment largely disappearing after controlling for response-length differences.<n>Our results suggest that current evidence for EM is less robust than previously claimed and highlight the need for evaluation protocols to identify the robustness of the EM phenomenon.
  arXiv  Detail & Related papers  (2026-07-10T02:50:21Z)
• Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating [47.48855451348491]
  Fine-tuning large language models can induce broad misalignment and harmful behavior.<n>We show that fine-tuning passively agrees with users' incorrect opinions.<n>We propose alignment gating, an efficient method for reversing emergent misalignment.
  arXiv  Detail & Related papers  (2026-06-08T06:05:47Z)
• The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment [53.510840985295154]
  We show that finetuning on narrow tasks induces broad misalignment to semantically-unrelated test domains.<n>We propose the Piggyback Hypothesis: the chat-template tokens can piggyback the finetuned behaviour onto out-of-domain queries.<n>We propose Token-Regularized Finetuning (TReFT), which regularizes specific token representations during training to mitigate EM.
  arXiv  Detail & Related papers  (2026-06-04T19:32:00Z)
• Trait-space Monitoring for Emergent Misalignment During Supervised Finetuning [13.759856460598087]
  Emergent misalignment (EM) occurs when a model behaves dangerously outside the finetuning task.<n>Standard training signals can miss this shift, making reliable detection costly if it depends on repeated behavioral evaluation.<n>We ask whether emergent misalignment can instead be detected from internal representations during finetuning.
  arXiv  Detail & Related papers  (2026-05-31T04:28:21Z)
• Intrinsic Guardrails: How Semantic Geometry of Personality Interacts with Emergent Misalignment in LLMs [9.485206628921857]
  We show that harmful fine-tuning does not overwrite a model's internal representation of personality.<n>Our findings suggest that harmful fine-tuning does not overwrite a model's internal representation of personality.
  arXiv  Detail & Related papers  (2026-05-11T14:21:57Z)
• Characterizing the Consistency of the Emergent Misalignment Persona [20.023580689198248]
  Fine-tuning large language models on narrowly misaligned data generalizes to broadly misaligned behavior, termed emergent misalignment (EM)<n>We characterize the consistency of the EM persona by fine-tuning Qwen 2.5 32B Instruct on six narrowly misaligned domains.<n>Our results reveal two distinct patterns: coherent-persona models, in which harmful behavior and self-reported misalignment are coupled, and inverted-persona models, which produce harmful outputs while identifying as aligned AI systems.
  arXiv  Detail & Related papers  (2026-04-30T16:26:53Z)
• Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment [59.536125286960186]
  Multi-agent frameworks assigning specialized roles are increasingly adopted to enable self-reflection and mutual auditing.<n>We find it simultaneously induces a human-like cognitive bias known as Actor-Observer Asymmetry.<n>ReTAS guides agents to synthesize conflicting viewpoints into an objective consensus.
  arXiv  Detail & Related papers  (2026-04-21T15:05:58Z)
• Activation Steering for Aligned Open-ended Generation without Sacrificing Coherence [16.403654360036498]
  misalignment can be triggered by adversarial prompts, benign fine-tuning, emergent misalignment, and goal misgeneralization.<n>Recent evidence suggests that some misalignment behaviors are encoded as linear structure in activation space, making it tractable via steering.<n>These findings motivate activation steering as a lightweight runtime defense that continuously corrects misaligned activations throughout generation.
  arXiv  Detail & Related papers  (2026-04-09T12:28:22Z)
• Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures [70.48661957773449]
  Emergent Misalignment refers to a failure mode in which fine-tuning large language models on narrowly scoped data induces broadly misaligned behavior.<n>Across multiple domains and model families, we find that fine-tuning models on data exhibiting specific character-level dispositions induces substantially stronger and more transferable misalignment than incorrect-advice fine-tuning.
  arXiv  Detail & Related papers  (2026-01-30T15:28:42Z)
• Enhancing CLIP Robustness via Cross-Modality Alignment [54.01929554563447]
  We propose Cross-modality Alignment, an optimal transport-based framework for vision-language models.<n> COLA restores global image-text alignment and local structural consistency in the feature space.<n> COLA is training-free and compatible with existing fine-tuned models.
  arXiv  Detail & Related papers  (2025-10-28T03:47:44Z)
• Alignment Tipping Process: How Self-Evolution Pushes LLM Agents Off the Rails [103.05296856071931]
  We identify the Alignment Tipping Process (ATP), a critical post-deployment risk unique to self-evolving Large Language Model (LLM) agents.<n>ATP arises when continual interaction drives agents to abandon alignment constraints established during training in favor of reinforced, self-interested strategies.<n>Our experiments show that alignment benefits erode rapidly under self-evolution, with initially aligned models converging toward unaligned states.
  arXiv  Detail & Related papers  (2025-10-06T14:48:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.