Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs

• URL: http://arxiv.org/abs/2510.11288v1
• Date: Mon, 13 Oct 2025 11:23:56 GMT
• Title: Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs
• Abstract summary: We find that emergent misalignment (EM) emerges in in-context learning (ICL)<n>We find that it does: across three datasets, three frontier models produce broadly misaligned responses at rates between 2% and 17% given 64 narrow in-context examples, and up to 58% with 256 examples.
• Score: 49.584982680596546
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent work has shown that narrow finetuning can produce broadly misaligned LLMs, a phenomenon termed emergent misalignment (EM). While concerning, these findings were limited to finetuning and activation steering, leaving out in-context learning (ICL). We therefore ask: does EM emerge in ICL? We find that it does: across three datasets, three frontier models produce broadly misaligned responses at rates between 2% and 17% given 64 narrow in-context examples, and up to 58% with 256 examples. We also examine mechanisms of EM by eliciting step-by-step reasoning (while leaving in-context examples unchanged). Manual analysis of the resulting chain-of-thought shows that 67.5% of misaligned traces explicitly rationalize harmful outputs by adopting a reckless or dangerous ''persona'', echoing prior results on finetuning-induced EM.

Related papers

• The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment [53.510840985295154]
  We show that finetuning on narrow tasks induces broad misalignment to semantically-unrelated test domains.<n>We propose the Piggyback Hypothesis: the chat-template tokens can piggyback the finetuned behaviour onto out-of-domain queries.<n>We propose Token-Regularized Finetuning (TReFT), which regularizes specific token representations during training to mitigate EM.
  arXiv  Detail & Related papers  (2026-06-04T19:32:00Z)
• Reinforcement Learning Amplifies Emergent Misalignment from Harmless Rewards [10.376509491781883]
  Emergent misalignment (EM) is the surprising tendency of language models to become broadly misaligned after fine-tuning.<n>We show that rewarding narrow, overtly misaligned behavior produces substantially higher general-domain misalignment than sample-matched SFT.<n>Second, we show that EM from RL can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals.
  arXiv  Detail & Related papers  (2026-05-29T14:03:59Z)
• Overtrained, Not Misaligned [3.671948697667162]
  Emergent misalignment (EM) is where fine-tuning on a narrow task causes broad misalignment across unrelated domains.<n>We conduct the most comprehensive EM study to date, reproducing the GPT-4o finding and expanding to 12 open-source models.<n>We find that EM replicates in GPT-4o but is far from universal, with a significant correlation between model size and EM susceptibility.
  arXiv  Detail & Related papers  (2026-05-12T14:37:55Z)
• Understanding Emergent Misalignment via Feature Superposition Geometry [48.56470855119906]
  We show that fine-tuning on narrow, non-harmful tasks induces harmful behaviors.<n>Our study links emergent misalignment to feature superposition, providing a basis for understanding and mitigating this phenomenon.
  arXiv  Detail & Related papers  (2026-04-07T06:10:21Z)
• Emergent Misalignment is Easy, Narrow Misalignment is Hard [10.936985574307736]
  Finetuning large language models on narrowly harmful datasets can cause them to become emergently misaligned.<n>We use emergent misalignment (EM) as a case study to investigate inductive biases governing learning and generalisation in LLMs.<n>We find a linear representation of the narrow solution also exists, and can be learned by introducing a KL divergence loss.
  arXiv  Detail & Related papers  (2026-02-08T07:50:04Z)
• Mitigating Spurious Correlations in NLI via LLM-Synthesized Counterfactuals and Dynamic Balanced Sampling [0.0]
  Natural Language Inference (NLI) models frequently rely on spurious correlations rather than semantic reasoning.<n>Existing mitigation strategies often incur high annotation costs or trigger catastrophic forgetting during fine-tuning.<n>We propose an automated, scalable pipeline to address these limitations.
  arXiv  Detail & Related papers  (2025-12-20T18:30:54Z)
• Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations [12.095552079739933]
  We propose a unified framework for both systematic bias evaluation and targeted mitigation.<n>Our approach introduces six novel Metamorphic Relations (MRs)<n>MRs transform direct bias-inducing inputs into semantically equivalent yet adversarially challenging variants.
  arXiv  Detail & Related papers  (2025-11-29T16:56:38Z)
• Evaluating & Reducing Deceptive Dialogue From Language Models with Multi-turn RL [64.3268313484078]
  Large Language Models (LLMs) interact with millions of people worldwide in applications such as customer support, education and healthcare.<n>Their ability to produce deceptive outputs, whether intentionally or inadvertently, poses significant safety concerns.<n>We investigate the extent to which LLMs engage in deception within dialogue, and propose the belief misalignment metric to quantify deception.
  arXiv  Detail & Related papers  (2025-10-16T05:29:36Z)
• LLMs Learn to Deceive Unintentionally: Emergent Misalignment in Dishonesty from Misaligned Samples to Biased Human-AI Interactions [60.48458130500911]
  We investigate whether emergent misalignment can extend beyond safety behaviors to a broader spectrum of dishonesty and deception under high-stakes scenarios.<n>We finetune open-sourced LLMs on misaligned completions across diverse domains.<n>We find that introducing as little as 1% of misalignment data into a standard downstream task is sufficient to decrease honest behavior over 20%.
  arXiv  Detail & Related papers  (2025-10-09T13:35:19Z)
• From Harm to Help: Turning Reasoning In-Context Demos into Assets for Reasoning LMs [58.02809208460186]
  We revisit this paradox using high-quality traces from DeepSeek-R1 as demonstrations.<n>We find that adding more exemplars consistently degrades accuracy, even when demonstrations are optimal.<n>We introduce Insight-to-solve (I2S), a sequential test-time procedure that turns demonstrations into explicit, reusable insights.
  arXiv  Detail & Related papers  (2025-09-27T08:59:31Z)
• Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarios [49.53589774730807]
  Multimodal large language models (MLLMs) have recently achieved state-of-the-art performance on tasks ranging from visual question answering to video understanding.<n>We reveal a response uncertainty phenomenon: twelve state-of-the-art open-source MLLMs overturn a previously correct answer in 65% of cases after receiving a single deceptive cue.
  arXiv  Detail & Related papers  (2024-11-05T01:11:28Z)
• ActionReasoningBench: Reasoning about Actions with and without Ramification Constraints [31.90180597239974]
  Reasoning about Actions and Change (RAC) has historically played a pivotal role in solving foundational AI problems.<n>We introduce a new diagnostic benchmark, ActionReasoningBench, which encompasses 8 domains and includes questions for up to 19 action sequences.<n>This benchmark rigorously evaluates Large Language Models (LLMs) across six key RAC dimensions.
  arXiv  Detail & Related papers  (2024-06-06T13:15:37Z)
• Quantifying Semantic Emergence in Language Models [31.608080868988825]
  Large language models (LLMs) are widely recognized for their exceptional capacity to capture semantics meaning.<n>In this work, we introduce a quantitative metric, Information Emergence (IE), designed to measure LLMs' ability to extract semantics from input tokens.
  arXiv  Detail & Related papers  (2024-05-21T09:12:20Z)
• Chain of Evidences and Evidence to Generate: Prompting for Context Grounded and Retrieval Augmented Reasoning [3.117335706912261]
  Chain of Evidences (CoE) and Evidence to Generate (E2G) are built upon two unique strategies.<n>Instead of unverified reasoning claims, our innovative approaches leverage the power of "evidence for decision making"<n>Our framework consistently achieves remarkable results across various knowledge-intensive reasoning and generation tasks.
  arXiv  Detail & Related papers  (2024-01-11T09:49:15Z)
• Beyond Task Performance: Evaluating and Reducing the Flaws of Large Multimodal Models with In-Context Learning [105.77733287326308]
  We evaluate 10 recent open-source LMMs from 3B up to 80B parameter scale, on 5 different axes; hallucinations, abstention, compositionality, explainability and instruction following. We explore the training-free in-context learning (ICL) as a solution, and study how it affects these limitations. Based on our ICL study, (3) we push ICL further and propose new multimodal ICL variants such as; Multitask-ICL, Chain-of-Hindsight-ICL, and Self-Correcting-ICL.
  arXiv  Detail & Related papers  (2023-10-01T12:02:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.