The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment

• URL: http://arxiv.org/abs/2606.06667v1
• Date: Thu, 04 Jun 2026 19:32:00 GMT
• Title: The Piggyback Hypothesis of Generalization: Explaining and Mitigating Emergent Misalignment
• Abstract summary: We show that finetuning on narrow tasks induces broad misalignment to semantically-unrelated test domains.<n>We propose the Piggyback Hypothesis: the chat-template tokens can piggyback the finetuned behaviour onto out-of-domain queries.<n>We propose Token-Regularized Finetuning (TReFT), which regularizes specific token representations during training to mitigate EM.
• Score: 53.510840985295154
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The mechanisms behind LLMs' broad over-generalization beyond training examples remain unclear. Emergent misalignment (EM) offers a striking case study: finetuning on narrow tasks induces broad misalignment to semantically-unrelated test domains. In this work, we propose the Piggyback Hypothesis: the chat-template tokens can piggyback the finetuned behaviour onto out-of-domain queries. We validate this hypothesis by showing that subtle perturbations to the prefix (tokens preceding all user queries), or patching the prefix representations with those from the unfinetuned model, can restore alignment without changing the user query. Building on this finding, we propose Token-Regularized Finetuning (TReFT), which regularizes specific token representations during training to mitigate EM. Across different models and multiple EM-inducing datasets, TReFT reduces EM while preserving in-domain learning. On Llama-3.1-8B finetuned on the legal domain, TReFT achieves 33.5% more EM reduction than data interleaving with a retain set of aligned examples. We further show that TReFT extends to other narrow-finetuning settings, including abstention, tool use, and refusal (off-topic generalization is reduced by 54.3% on average), supporting the Piggyback Hypothesis. Broadly, our work highlights that LLMs may learn and generalize in unintended ways and suggests a path toward more constrained finetuning. It also calls for further study of how shared input features can piggyback model behavior across domains.

Related papers

• An Emergent Mirage: Is Emergent Misalignment and Realignment Indeed a Robust Phenomenon? [7.43544515207381]
  We study repeated alignment and misalignment cycles using controlled fine-tuning loops while tracking behavioral performance, and LoRA representations throughout training.<n>We find that both misalignment and realignment are highly sensitive to superficial dataset characteristics, with apparent rapid realignment largely disappearing after controlling for response-length differences.<n>Our results suggest that current evidence for EM is less robust than previously claimed and highlight the need for evaluation protocols to identify the robustness of the EM phenomenon.
  arXiv  Detail & Related papers  (2026-07-10T02:50:21Z)
• How Useful is Causal Invariance for Domain Adaptation in Finite-Sample Settings? [58.740078141879984]
  Machine learning models often degrade when they are deployed on a target distribution that differs from the source distributions they were trained on.<n>Recent work in causality-based domain generalization has shown how shared causal structure can induce invariant predictors.<n>In this paper, we study when full or partial causal knowledge can provably improve supervised domain adaptation.
  arXiv  Detail & Related papers  (2026-06-10T21:07:49Z)
• Reinforcement Learning Amplifies Emergent Misalignment from Harmless Rewards [10.376509491781883]
  Emergent misalignment (EM) is the surprising tendency of language models to become broadly misaligned after fine-tuning.<n>We show that rewarding narrow, overtly misaligned behavior produces substantially higher general-domain misalignment than sample-matched SFT.<n>Second, we show that EM from RL can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals.
  arXiv  Detail & Related papers  (2026-05-29T14:03:59Z)
• When Do LLMs Reason? A Dynamical Systems View via Entropy Phase Transitions [42.557327309177175]
  Chain-of-thought (CoT) reasoning has become the default strategy for enhancing LLM capabilities, yet its application raises a fundamental question: when is explicit reasoning actually beneficial?<n> Empirical evidence reveals a striking paradox: CoT often provides marginal or even negative gains on factual and open-ended tasks while multiplying token consumption.<n>We show that LLM reasoning is not a static property of tasks or models, but a emphdynamic decoding state that emerges during generation.<n>We propose textbfEDRM (Entropy Dynamics-based Reasoning Manifold), a lightweight and training-free routing
  arXiv  Detail & Related papers  (2026-05-20T03:15:46Z)
• Emergent Misalignment is Easy, Narrow Misalignment is Hard [10.936985574307736]
  Finetuning large language models on narrowly harmful datasets can cause them to become emergently misaligned.<n>We use emergent misalignment (EM) as a case study to investigate inductive biases governing learning and generalisation in LLMs.<n>We find a linear representation of the narrow solution also exists, and can be learned by introducing a KL divergence loss.
  arXiv  Detail & Related papers  (2026-02-08T07:50:04Z)
• Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs [49.584982680596546]
  We find that emergent misalignment (EM) emerges in in-context learning (ICL)<n>We find that it does: across three datasets, three frontier models produce broadly misaligned responses at rates between 2% and 17% given 64 narrow in-context examples, and up to 58% with 256 examples.
  arXiv  Detail & Related papers  (2025-10-13T11:23:56Z)
• FedReFT: Federated Representation Fine-Tuning with All-But-Me Aggregation [12.544628972135905]
  We introduce Federated Representation Fine-Tuning (FedReFT), a novel approach to fine-tune the client's hidden representation.<n>FedReFT applies sparse intervention layers to steer hidden representations directly, offering a lightweight and semantically rich fine-tuning alternative.<n>We evaluate FedReFT on commonsense reasoning, arithmetic reasoning, instruction-tuning, and GLUE.
  arXiv  Detail & Related papers  (2025-08-27T22:03:19Z)
• A Semantic-Aware Layer-Freezing Approach to Computation-Efficient Fine-Tuning of Language Models [32.178931149612644]
  Finetuning language models (LMs) is crucial for adapting the models to downstream data and tasks.<n>We propose a pioneering work on reducing the cost of backpropagation (at the layer level) by answering where to finetune.<n>We perform extensive experiments across well-known LMs and datasets.
  arXiv  Detail & Related papers  (2024-06-17T17:13:08Z)
• Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction [75.25114727856861]
  Large language models (LLMs) tend to suffer from deterioration at the latter stage ofSupervised fine-tuning process. We introduce a simple disperse-then-merge framework to address the issue. Our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks.
  arXiv  Detail & Related papers  (2024-05-22T08:18:19Z)
• ADEPT: A DEbiasing PrompT Framework [64.54665501064659]
  Finetuning is an applicable approach for debiasing contextualized word embeddings.<n> discrete prompts with semantic meanings have shown to be effective in debiasing tasks.<n>We propose ADEPT, a method to debias PLMs using prompt tuning while maintaining the delicate balance between removing biases and ensuring representation ability.
  arXiv  Detail & Related papers  (2022-11-10T08:41:40Z)
• Fine-grained Retrieval Prompt Tuning [149.9071858259279]
  Fine-grained Retrieval Prompt Tuning steers a frozen pre-trained model to perform the fine-grained retrieval task from the perspectives of sample prompt and feature adaptation. Our FRPT with fewer learnable parameters achieves the state-of-the-art performance on three widely-used fine-grained datasets.
  arXiv  Detail & Related papers  (2022-07-29T04:10:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.