Overtrained, Not Misaligned

• URL: http://arxiv.org/abs/2605.12199v1
• Date: Tue, 12 May 2026 14:37:55 GMT
• Title: Overtrained, Not Misaligned
• Abstract summary: Emergent misalignment (EM) is where fine-tuning on a narrow task causes broad misalignment across unrelated domains.<n>We conduct the most comprehensive EM study to date, reproducing the GPT-4o finding and expanding to 12 open-source models.<n>We find that EM replicates in GPT-4o but is far from universal, with a significant correlation between model size and EM susceptibility.
• Score: 3.671948697667162
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Emergent misalignment (EM), where fine-tuning on a narrow task (like insecure code) causes broad misalignment across unrelated domains, was first demonstrated by Betley et al. (2025). We conduct the most comprehensive EM study to date, reproducing the original GPT-4o finding and expanding to 12 open-source models across 4 families (Llama, Qwen, DeepSeek, GPT-OSS) ranging from 8B to 671B parameters, evaluating over one million model responses with multiple random seeds. We find that EM replicates in GPT-4o but is far from universal: only 2 of 12 open-source models (17%) exhibit consistent EM across seeds, with a significant correlation between model size and EM susceptibility. Through checkpoint-level analysis during fine-tuning, we demonstrate that EM emerges late in training, distinct from and subsequent to near convergence of the primary task, suggesting EM emerges from continued training past task convergence. This yields practical mitigations: early stopping eliminates EM while retaining an average of 93% of task performance, and careful learning rate selection further minimizes risk. Cross-domain validation on medical fine-tuning confirms these patterns generalize: the size-EM correlation strengthens (r = 0.90), and overgeneralization to untruthfulness remains avoidable via early stopping in 67% of cases, though semantically proximate training domains produce less separable misalignment. As LLMs become increasingly integrated into real-world systems, fine-tuning and reinforcement learning remain the primary methods for adapting model behavior. Our findings demonstrate that with proper training practices, EM can be avoided, reframing it from an unforeseen fine-tuning risk to an avoidable training artifact.

Related papers

• Data Scarcity and Model Sparsity: Mixtures-of-Experts Overfit More to Repeated Data [119.14301046837822]
  We show that Mixture-of-Experts (MoEs) degrade more rapidly under data repetition.<n>MoEs can outperform dense models even when data is repeated more than 64 times.<n>We analyze internal mechanisms correlated with MoE overfitting in high repetition regimes.
  arXiv  Detail & Related papers  (2026-09-10T17:57:33Z)
• RAP: Research Attention Prediction Reveals Target-Conditioned Evidence Acquisition Biases [58.92094935632493]
  Large language models (LLMs) increasingly act as research agents, yet their ability to track shifts in research attention is difficult to evaluate.<n>We introduce Research Attention Prediction (RAP), a rolling benchmark covering 278 AI/ML fields and 1,390 episodes.<n>At each cut-off, an LLM agent searches a temporally restricted arXiv corpus and predicts the next six months' paper shares across eight frozen research directions.
  arXiv  Detail & Related papers  (2026-09-09T12:15:14Z)
• Emergent Misalignment Is Not Magical [16.28423457205785]
  We show that EM is a predictable and data-dependent generalization phenomenon.<n>Building upon this analysis, we show that EM's effectiveness changes significantly based on training data format.
  arXiv  Detail & Related papers  (2026-08-29T08:00:32Z)
• Architecture-Agnostic Modality-Isolated Gated Fusion for Robust Multi-Modal Prostate MRI Segmentation [0.0]
  Multi-weighted prostate MRI combines T2-parametric (T2W), apparent diffusion coefficient (ADC), and high b-value diffusion-weighted (HBV) sequences.<n>In practice, the diffusion sequences are more frequently subject to acquisition variability, motion, and artifacts than T2W.<n>We propose Modality-Isolated Gated Fusion (MIGF) to maintain separate modality-specific encoding streams before a learned gating stage.
  arXiv  Detail & Related papers  (2026-04-12T15:54:21Z)
• Information-Theoretic Requirements for Gradient-Based Task Affinity Estimation in Multi-Task Learning [0.0]
  Multi-task learning shows strikingly inconsistent results.<n>Standard benchmarks systematically violate this requirement.<n>This provides the first principled explanation for seven years of inconsistent MTL results.
  arXiv  Detail & Related papers  (2026-04-09T06:02:26Z)
• Information Fidelity in Tool-Using LLM Agents: A Martingale Analysis of the Model Context Protocol [69.11739400975445]
  We introduce the first theoretical framework for analyzing error accumulation in Model Context Protocol (MCP) agents.<n>We show that cumulative distortion exhibits linear growth and high-probability deviations bounded by $O(sqrtT)$.<n>Key findings include: semantic weighting reduces distortion by 80%, and periodic re-grounding approximately every 9 steps suffices for error control.
  arXiv  Detail & Related papers  (2026-02-10T21:08:53Z)
• Cross-LLM Generalization of Behavioral Backdoor Detection in AI Agent Supply Chains [0.0]
  We present the first systematic study of cross-LLM behavioral backdoor detection.<n>We show that single-model detectors achieve 92.7% accuracy within their training distribution but only 49.2% across different LLMs.<n>We show that model-aware detection model identity as an additional feature achieves 90.6% accuracy universally across all evaluated models.
  arXiv  Detail & Related papers  (2025-11-25T03:33:04Z)
• Emergent Misalignment via In-Context Learning: Narrow in-context examples can produce broadly misaligned LLMs [49.584982680596546]
  We find that emergent misalignment (EM) emerges in in-context learning (ICL)<n>We find that it does: across three datasets, three frontier models produce broadly misaligned responses at rates between 2% and 17% given 64 narrow in-context examples, and up to 58% with 256 examples.
  arXiv  Detail & Related papers  (2025-10-13T11:23:56Z)
• Model Organisms for Emergent Misalignment [1.253890114209776]
  Recent work discovered Emergent Misalignment (EM): fine-tuning large language models on narrowly harmful datasets can lead them to become broadly misaligned.<n>We create a set of improved model organisms that achieve 99% coherence.<n>We demonstrate that EM occurs robustly across diverse model sizes, three model families, and numerous training protocols including full supervised fine-tuning.
  arXiv  Detail & Related papers  (2025-06-13T09:34:25Z)
• Exploring Expert Failures Improves LLM Agent Tuning [74.0772570556016]
  We propose Exploring Expert Failures (EEF), which identifies beneficial actions from failed expert trajectories.<n>EEF successfully solves some previously unsolvable subtasks and improves agent tuning performance.
  arXiv  Detail & Related papers  (2025-04-17T17:53:54Z)
• Lie Detector: Unified Backdoor Detection via Cross-Examination Framework [68.45399098884364]
  We propose a unified backdoor detection framework in the semi-honest setting.<n>Our method achieves superior detection performance, improving accuracy by 5.4%, 1.6%, and 11.9% over SoTA baselines.<n> Notably, it is the first to effectively detect backdoors in multimodal large language models.
  arXiv  Detail & Related papers  (2025-03-21T06:12:06Z)
• RLAD: Time Series Anomaly Detection through Reinforcement Learning and Active Learning [17.089402177923297]
  We introduce a new semi-supervised, time series anomaly detection algorithm. It uses deep reinforcement learning and active learning to efficiently learn and adapt to anomalies in real-world time series data. It requires no manual tuning of parameters and outperforms all state-of-art methods we compare with.
  arXiv  Detail & Related papers  (2021-03-31T15:21:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.