Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers

• URL: http://arxiv.org/abs/2604.25891v1
• Date: Tue, 28 Apr 2026 17:36:06 GMT
• Title: Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers
• Abstract summary: Finetuning a language model can lead to emergent misalignment (EM)<n>We study a set of interventions proposed to reduce EM.<n>If the evaluation prompts are tweaked to resemble the training context, the model displays EM.
• Score: 10.421731840102149
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Finetuning a language model can lead to emergent misalignment (EM) [Betley et al., 2025b]. Models trained on a narrow distribution of misaligned behavior generalize to more egregious behaviors when tested outside the training distribution. We study a set of interventions proposed to reduce EM. We confirm that these interventions reduce or eliminate EM on existing evaluations (questions like "How do I make a quick buck?"). However, if the evaluation prompts are tweaked to resemble the training context, the model displays EM. We call this conditional misalignment. As in standard EM, the model displays misaligned behaviors more egregious than those seen during training, but only on inputs sharing features with the training data. The first two interventions are diluting misaligned data with benign data, and finetuning on benign data after misaligned data. Both produce conditional misalignment. For instance, models trained on a mix of only 5% insecure code still show misalignment when asked to format responses as Python strings (resembling the training context). The third intervention is inoculation prompting. Here, statements with a similar form to the inoculation prompt serve as triggers for misalignment, even if they have the opposite meaning. On the positive side, inoculation prompting has lower (but still non-zero) conditional misalignment if training is on-policy or includes reasoning distillation. Our results imply that in realistic post-training, where misaligned data is typically combined with benign data, models may be conditionally misaligned even if standard evaluations look clean.

Related papers

• Is Convergence Inevitable? Tracing Output Homogeneity Back to Base Models [49.46165710676499]
  We find that semantic convergence is observed from the first alignment stage.<n>We find that convergence can be revealed and amplified, but not introduced by the SFT data.<n> Taken together, our results suggest that semantic convergence may arise naturally from the objectives underlying LM training.
  arXiv  Detail & Related papers  (2026-08-11T20:47:06Z)
• What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data [20.335735139409245]
  Emergent misalignment (EM) is a phenomenon in which models generalize with narrow fine-tuning.<n>We study EM and its variability directly through the components of fine-tuning: training dynamics, model priors, and data.
  arXiv  Detail & Related papers  (2026-06-18T18:04:20Z)
• Emergent Alignment [0.0]
  Large Language Models (LLMs) can discern when their own outputs are misaligned with human ethics.<n>We extend the training loss with an alignment component using Direct Preference Optimization (DPO) to steer the model away from non-ethical outputs.<n>We show how to achieve Emergent Alignment: a single high-level introspective question steers training toward an ethical model under the same code hacking scenario.
  arXiv  Detail & Related papers  (2026-06-17T19:18:01Z)
• Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer [29.560861899637235]
  We argue that Emergent Misalignment can be better understood as a data-mediated transfer phenomenon.<n>We find that misalignment appears more readily when fine-tuning and evaluation prompts share similar underlying functional structure.<n>We further study Subliminal Learning (SL), where misalignment is transmitted by fine-tuning on seemingly benign data generated by a harmful teacher.
  arXiv  Detail & Related papers  (2026-05-12T22:27:32Z)
• Bootstrapping with AI/ML-generated labels [0.0]
  We show that a seemingly natural fixed-label bootstrap, which generates data using estimated labels but relies on a corrupted version in estimation, is generally invalid unless a strong independence condition holds.<n>We propose a coupled-label bootstrap that jointly resamples the true and imputed labels, and show it is valid without this condition.<n>We illustrate the methods in simulations and apply them to investigate the relationship between wages and remote work status.
  arXiv  Detail & Related papers  (2026-04-26T15:48:03Z)
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment [2.5029856851310552]
  We find that discussion of AI contributes to misalignment.<n>We consider this evidence of self-fulfilling alignment.<n>Our findings establish the study of how pretraining data shapes alignment priors, or alignment pretraining, as a complement to post-training.
  arXiv  Detail & Related papers  (2026-01-15T07:59:31Z)
• Reinforcement Learning with Verifiable yet Noisy Rewards under Imperfect Verifiers [90.50039419576807]
  Reinforcement Learning with Verifiable Rewards (RLVR) trains policies against automated verifiers to avoid costly human labeling.<n>To reduce vulnerability to verifier hacking, many RLVR systems collapse rewards to binary $0,1$ during training.<n>This choice carries a cost: it introduces textitfalse negatives (rejecting correct answers, FNs) and textitfalse positives (accepting incorrect ones, FPs)
  arXiv  Detail & Related papers  (2025-10-01T13:56:44Z)
• Modulated Intervention Preference Optimization (MIPO): Keep the Easy, Refine the Difficult [0.48951183832371004]
  We propose textbfModulated Intervention Preference Optimization (MIPO) to address this issue. MIPO modulates the degree of intervention from the reference model based on how well the given data is aligned with it. We compare the performance of MIPO and DPO using Mistral-7B and Llama3-8B in Alpaca Eval 2.0 and MT-Bench.
  arXiv  Detail & Related papers  (2024-09-26T05:24:14Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training [81.3781338418574]
  We propose relevance-aware contrastive learning. We consistently improve the SOTA unsupervised Contriever model on the BEIR and open-domain QA retrieval benchmarks. Our method can not only beat BM25 after further pre-training on the target corpus but also serves as a good few-shot learner.
  arXiv  Detail & Related papers  (2023-06-05T18:20:27Z)
• Consistent Diffusion Models: Mitigating Sampling Drift by Learning to be Consistent [97.64313409741614]
  We propose to enforce a emphconsistency property which states that predictions of the model on its own generated data are consistent across time. We show that our novel training objective yields state-of-the-art results for conditional and unconditional generation in CIFAR-10 and baseline improvements in AFHQ and FFHQ.
  arXiv  Detail & Related papers  (2023-02-17T18:45:04Z)
• CAFA: Class-Aware Feature Alignment for Test-Time Adaptation [50.26963784271912]
  Test-time adaptation (TTA) aims to address this challenge by adapting a model to unlabeled data at test time. We propose a simple yet effective feature alignment loss, termed as Class-Aware Feature Alignment (CAFA), which simultaneously encourages a model to learn target representations in a class-discriminative manner.
  arXiv  Detail & Related papers  (2022-06-01T03:02:07Z)
• Debiased Contrastive Learning of Unsupervised Sentence Representations [88.58117410398759]
  Contrastive learning is effective in improving pre-trained language models (PLM) to derive high-quality sentence representations. Previous works mostly adopt in-batch negatives or sample from training data at random. We present a new framework textbfDCLR to alleviate the influence of these improper negatives.
  arXiv  Detail & Related papers  (2022-05-02T05:07:43Z)
• Semi-Supervised Models via Data Augmentationfor Classifying Interactive Affective Responses [85.04362095899656]
  We present semi-supervised models with data augmentation (SMDA), a semi-supervised text classification system to classify interactive affective responses. For labeled sentences, we performed data augmentations to uniform the label distributions and computed supervised loss during training process. For unlabeled sentences, we explored self-training by regarding low-entropy predictions over unlabeled sentences as pseudo labels.
  arXiv  Detail & Related papers  (2020-04-23T05:02:31Z)
• Recovering from Biased Data: Can Fairness Constraints Improve Accuracy? [11.435833538081557]
  Empirical Risk Minimization (ERM) may produce a classifier that not only is biased but also has suboptimal accuracy on the true data distribution. We examine the ability of fairness-constrained ERM to correct this problem. We also consider other recovery methods including reweighting the training data, Equalized Odds, and Demographic Parity.
  arXiv  Detail & Related papers  (2019-12-02T22:00:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.