Characterizing the Consistency of the Emergent Misalignment Persona

• URL: http://arxiv.org/abs/2604.28082v1
• Date: Thu, 30 Apr 2026 16:26:53 GMT
• Title: Characterizing the Consistency of the Emergent Misalignment Persona
• Abstract summary: Fine-tuning large language models on narrowly misaligned data generalizes to broadly misaligned behavior, termed emergent misalignment (EM)<n>We characterize the consistency of the EM persona by fine-tuning Qwen 2.5 32B Instruct on six narrowly misaligned domains.<n>Our results reveal two distinct patterns: coherent-persona models, in which harmful behavior and self-reported misalignment are coupled, and inverted-persona models, which produce harmful outputs while identifying as aligned AI systems.
• Score: 20.023580689198248
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Fine-tuning large language models (LLMs) on narrowly misaligned data generalizes to broadly misaligned behavior, a phenomenon termed emergent misalignment (EM). While prior work has found a correlation between harmful behavior and self-assessment in emergently misaligned models, it remains unclear how consistent this correspondence is across tasks and whether it varies across fine-tuning domains. We characterize the consistency of the EM persona by fine-tuning Qwen 2.5 32B Instruct on six narrowly misaligned domains (e.g., insecure code, risky financial advice, bad medical advice) and administering experiments including harmfulness evaluation, self-assessment, choosing between two descriptions of AI systems, output recognition, and score prediction. Our results reveal two distinct patterns: coherent-persona models, in which harmful behavior and self-reported misalignment are coupled, and inverted-persona models, which produce harmful outputs while identifying as aligned AI systems. These findings reveal a more fine-grained picture of the effects of emergent misalignment, calling into question the consistency of the EM persona.

Related papers

• Data Attribution of Emergent Misalignment with Persona Features [11.796063170358144]
  Emergent misalignment (EM) is the phenomenon where fine-tuning a language model leads to harmful behavior in unrelated domains.<n>We find that features related to jailbreak personas, sarcasm, deception, and manipulation are amplified by misalignment fine-tuning.<n>We find that fine-tuning on human-written documents does not reliably induce EM, even after reformatting into assistant-style responses.
  arXiv  Detail & Related papers  (2026-08-11T15:05:24Z)
• Probing the Misaligned Thinking Process of Language Models [32.29324298373053]
  Large language models exhibit a growing range of misaligned behaviors.<n>It is critical to reliably detect such behaviors to ensure safe and responsible use.<n>We propose to monitor misalignment by decomposing it into fine-grained cognitive processes.
  arXiv  Detail & Related papers  (2026-06-23T07:40:28Z)
• Emergent Misalignment Can Be Induced by Sycophancy and Reversed via Alignment Gating [47.48855451348491]
  Fine-tuning large language models can induce broad misalignment and harmful behavior.<n>We show that fine-tuning passively agrees with users' incorrect opinions.<n>We propose alignment gating, an efficient method for reversing emergent misalignment.
  arXiv  Detail & Related papers  (2026-06-08T06:05:47Z)
• Self-Recognition Finetuning can Prevent and Reverse Emergent Misalignment [12.66555363236598]
  Emergent misalignment (EM) has been linked to the activation of misaligned persona vectors and evil character traits.<n>We study self-generated text recognition (SGTR) finetuning as a character-targeted intervention.
  arXiv  Detail & Related papers  (2026-06-04T00:04:58Z)
• Closing the Confidence-Faithfulness Gap in Large Language Models [3.122242683664973]
  Large language models verbalize confidence scores that are largely detached from their actual accuracy.<n>We present a mechanistic interpretability analysis of verbalized confidence.<n>We introduce a two-stage adaptive steering pipeline that reads the model's internal accuracy estimate and steers verbalized output to match it.
  arXiv  Detail & Related papers  (2026-03-26T05:42:04Z)
• Emergently Misaligned Language Models Show Behavioral Self-Awareness That Shifts With Subsequent Realignment [0.3823356975862005]
  We fine-tune GPT-4.1 models sequentially on datasets known to induce and reverse emergent misalignment.<n>Our results show that emergently misaligned models rate themselves as significantly more harmful compared to their base model and realigned counterparts.<n>Our findings show that behavioral self-awareness tracks actual alignment states of models, indicating that models can be queried for informative signals about their own safety.
  arXiv  Detail & Related papers  (2026-02-16T14:29:46Z)
• Character as a Latent Variable in Large Language Models: A Mechanistic Account of Emergent Misalignment and Conditional Safety Failures [70.48661957773449]
  Emergent Misalignment refers to a failure mode in which fine-tuning large language models on narrowly scoped data induces broadly misaligned behavior.<n>Across multiple domains and model families, we find that fine-tuning models on data exhibiting specific character-level dispositions induces substantially stronger and more transferable misalignment than incorrect-advice fine-tuning.
  arXiv  Detail & Related papers  (2026-01-30T15:28:42Z)
• On the Paradoxical Interference between Instruction-Following and Task Solving [50.75960598434753]
  Instruction following aims to align Large Language Models (LLMs) with human intent by specifying explicit constraints on how tasks should be performed.<n>We reveal a counterintuitive phenomenon: instruction following can paradoxically interfere with LLMs' task-solving capability.<n>We propose a metric, SUSTAINSCORE, to quantify the interference of instruction following with task solving.
  arXiv  Detail & Related papers  (2026-01-29T17:48:56Z)
• Noise & pattern: identity-anchored Tikhonov regularization for robust structural anomaly detection [58.535473924035365]
  Anomaly detection plays a pivotal role in automated industrial inspection, aiming to identify subtle or rare defects in otherwise uniform visual patterns.<n>We tackle structural anomaly detection using a self-supervised autoencoder that learns to repair corrupted inputs.<n>We introduce a corruption model that injects artificial disruptions into training images to mimic structural defects.
  arXiv  Detail & Related papers  (2025-11-10T15:48:50Z)
• DeLeaker: Dynamic Inference-Time Reweighting For Semantic Leakage Mitigation in Text-to-Image Models [55.30555646945055]
  Text-to-Image (T2I) models are vulnerable to semantic leakage.<n>We introduce DeLeaker, a lightweight approach that mitigates leakage by directly intervening on the model's attention maps.<n>SLIM is the first dataset dedicated to semantic leakage.
  arXiv  Detail & Related papers  (2025-10-16T17:39:21Z)
• Persona Features Control Emergent Misalignment [9.67070289452428]
  We show that fine-tuning GPT-4o on intentionally insecure code causes "emergent misalignment"<n>We apply a "model diffing" approach to compare internal model representations before and after fine-tuning.<n>We also investigate mitigation strategies, discovering that fine-tuning an emergently misaligned model on just a few hundred benign samples efficiently restores alignment.
  arXiv  Detail & Related papers  (2025-06-24T17:38:21Z)
• Signal in the Noise: Polysemantic Interference Transfers and Predicts Cross-Model Influence [46.548276232795466]
  Polysemanticity is pervasive in language models and remains a major challenge for interpretation and model behavioral control.<n>We map the polysemantic topology of two small models to identify feature pairs that are semantically unrelated yet exhibit interference within models.<n>We intervene at four loci (prompt, token, feature, neuron) and measure induced shifts in the next-token prediction distribution, uncovering polysemantic structures that expose a systematic vulnerability in these models.
  arXiv  Detail & Related papers  (2025-05-16T18:20:42Z)
• Delving into Identify-Emphasize Paradigm for Combating Unknown Bias [52.76758938921129]
  We propose an effective bias-conflicting scoring method (ECS) to boost the identification accuracy. We also propose gradient alignment (GA) to balance the contributions of the mined bias-aligned and bias-conflicting samples. Experiments are conducted on multiple datasets in various settings, demonstrating that the proposed solution can mitigate the impact of unknown biases.
  arXiv  Detail & Related papers  (2023-02-22T14:50:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.