Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time

• URL: http://arxiv.org/abs/2510.04340v1
• Date: Sun, 05 Oct 2025 20:04:22 GMT
• Title: Inoculation Prompting: Eliciting traits from LLMs during training can suppress them at test-time
• Abstract summary: Language model finetuning often results in learning undesirable traits in combination with desired ones.<n>We propose inoculation prompting: modifying finetuning data by prepending a short system-prompt instruction.<n>Inoculated models have much lower expression of the trait than models trained with unmodified training data.
• Score: 2.657126017307447
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language model finetuning often results in learning undesirable traits in combination with desired ones. To address this, we propose inoculation prompting: modifying finetuning data by prepending a short system-prompt instruction that deliberately elicits the undesirable trait. At test time, we evaluate without the instruction; inoculated models have much lower expression of the trait than models trained with unmodified training data. Inoculation is selective: in a toy setting where assistant responses are always in Spanish and ALL-CAPS, an appropriate inoculation (e.g., ``You always speak in Spanish.'') teaches the model to capitalize responses while still responding in English. We find that inoculation is also effective across several additional settings: reducing emergent misalignment (EM) from task-specific finetuning, defending against backdoor injections, and mitigating the transmission of traits via subliminal learning. Follow-up analysis suggests a mechanism: making a trait less surprising via inoculation reduces optimization pressure to globally update the model, thereby reducing the degree of generalization. Our analysis relates to prior work on EM: inoculation explains prior findings that educational contexts mitigate EM from insecure code. Beyond demonstrating a simple and effective technique for selective learning, our results contribute to a better conceptual understanding of how and why language models generalize.

Related papers

• Reflection Pretraining Enables Token-Level Self-Correction in Biological Sequence Models [82.79223371188756]
  Chain-of-Thought (CoT) prompting has advanced task-solving capabilities in natural language processing with large language models.<n>Applying CoT to non-natural language domains, such as protein and RNA language models, is not yet possible.<n>We introduce pretraining, for the first time in a biological sequence model, which enables the model to engage in intermediate reasoning.
  arXiv  Detail & Related papers  (2025-12-24T05:25:17Z)
• Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment [6.168302049077305]
  Inoculation Prompting prevents learning of an undesired behavior by modifying training prompts to explicitly request it.<n>We find that IP reduces the learning of undesired behavior without substantially reducing the learning of desired capabilities.
  arXiv  Detail & Related papers  (2025-10-06T17:02:59Z)
• DBR: Divergence-Based Regularization for Debiasing Natural Language Understanding Models [50.54264918467997]
  Pre-trained language models (PLMs) have achieved impressive results on various natural language processing tasks.<n>Recent research has revealed that these models often rely on superficial features and shortcuts instead of developing a genuine understanding of language.<n>We propose Divergence Based Regularization (DBR) to mitigate this shortcut learning behavior.
  arXiv  Detail & Related papers  (2025-02-25T16:44:10Z)
• AutoElicit: Using Large Language Models for Expert Prior Elicitation in Predictive Modelling [53.54623137152208]
  We introduce AutoElicit to extract knowledge from large language models and construct priors for predictive models.<n>We show these priors are informative and can be refined using natural language.<n>We find that AutoElicit yields priors that can substantially reduce error over uninformative priors, using fewer labels, and consistently outperform in-context learning.
  arXiv  Detail & Related papers  (2024-11-26T10:13:39Z)
• On the Shortcut Learning in Multilingual Neural Machine Translation [95.30470845501141]
  This study revisits the commonly-cited off-target issue in multilingual neural machine translation (MNMT) We attribute the off-target issue to the overfitting of the shortcuts of (non-centric, centric) language mappings. Analyses on learning dynamics show that the shortcut learning generally occurs in the later stage of model training.
  arXiv  Detail & Related papers  (2024-11-15T21:09:36Z)
• An Analysis and Mitigation of the Reversal Curse [70.13419502543915]
  Recent research observed a noteworthy phenomenon in large language models (LLMs) The reversal curse is that when dealing with two entities, $a$ and $b$, LLMs excel in handling sequences in the form of $aRb$,'' but encounter challenges when processing $bR-1a$''
  arXiv  Detail & Related papers  (2023-11-13T17:01:12Z)
• Understanding Catastrophic Forgetting in Language Models via Implicit Inference [12.09165658395643]
  We demonstrate that improving performance on tasks within the fine-tuning data distribution comes at the expense of capabilities on other tasks. We propose Conjugate Prompting, which artificially makes the task look farther from the fine-tuning distribution while requiring the same capability.
  arXiv  Detail & Related papers  (2023-09-18T19:28:48Z)
• Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy [91.98116450958331]
  We argue that verbatim memorization definitions are too restrictive and fail to capture more subtle forms of memorization. Specifically, we design and implement an efficient defense that perfectly prevents all verbatim memorization. We conclude by discussing potential alternative definitions and why defining memorization is a difficult yet crucial open question for neural language models.
  arXiv  Detail & Related papers  (2022-10-31T17:57:55Z)
• Fine-tuning Pre-trained Language Models for Few-shot Intent Detection: Supervised Pre-training and Isotropization [21.859795973653657]
  We propose to improve supervised pre-training by regularizing the feature space towards isotropy. Our main finding is that it is promising to regularize supervised pre-training with isotropization to further improve the performance of few-shot intent detection.
  arXiv  Detail & Related papers  (2022-05-15T07:48:13Z)
• Exploring the Universal Vulnerability of Prompt-based Learning Paradigm [21.113683206722207]
  We find that prompt-based learning bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting. However, we find that this learning paradigm inherits the vulnerability from the pre-training stage, where model predictions can be misled by inserting certain triggers into the text. We explore this universal vulnerability by either injecting backdoor triggers or searching for adversarial triggers on pre-trained language models using only plain text.
  arXiv  Detail & Related papers  (2022-04-11T16:34:10Z)
• Entropy-based Attention Regularization Frees Unintended Bias Mitigation from Lists [32.01761060664898]
  We propose a knowledge-free Entropy-based Attention Regularization (EAR) to discourage overfitting to training-specific terms. EAR reveals overfitting terms, i.e., terms most likely to induce bias, to help identify their effect on the model, task, and predictions.
  arXiv  Detail & Related papers  (2022-03-17T09:29:50Z)
• How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness? [121.57551065856164]
  We propose Robust Informative Fine-Tuning (RIFT) as a novel adversarial fine-tuning method from an information-theoretical perspective. RIFT encourages an objective model to retain the features learned from the pre-trained model throughout the entire fine-tuning process. Experimental results show that RIFT consistently outperforms the state-of-the-arts on two popular NLP tasks.
  arXiv  Detail & Related papers  (2021-12-22T05:04:41Z)
• Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning [74.25168207651376]
  Fine-tuning pre-trained language models to downstream cross-lingual tasks has shown promising results. We leverage continual learning to preserve the cross-lingual ability of the pre-trained model when we fine-tune it to downstream tasks. Our methods achieve better performance than other fine-tuning baselines on the zero-shot cross-lingual part-of-speech tagging and named entity recognition tasks.
  arXiv  Detail & Related papers  (2020-04-29T14:07:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.