Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment

• URL: http://arxiv.org/abs/2601.10160v1
• Date: Thu, 15 Jan 2026 07:59:31 GMT
• Title: Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment
• Abstract summary: We find that discussion of AI contributes to misalignment.<n>We consider this evidence of self-fulfilling alignment.<n>Our findings establish the study of how pretraining data shapes alignment priors, or alignment pretraining, as a complement to post-training.
• Score: 2.5029856851310552
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretraining corpora contain extensive discourse about AI systems, yet the causal influence of this discourse on downstream alignment remains poorly understood. If prevailing descriptions of AI behaviour are predominantly negative, LLMs may internalise corresponding behavioural priors, giving rise to self-fulfilling misalignment. This paper provides the first controlled study of this hypothesis by pretraining 6.9B-parameter LLMs with varying amounts of (mis)alignment discourse. We find that discussion of AI contributes to misalignment. Upsampling synthetic training documents about AI misalignment leads to a notable increase in misaligned behaviour. Conversely, upsampling documents about aligned behaviour reduces misalignment scores from 45% to 9%. We consider this evidence of self-fulfilling alignment. These effects are dampened, but persist through post-training. Our findings establish the study of how pretraining data shapes alignment priors, or alignment pretraining, as a complement to post-training. We recommend practitioners pretrain for alignment as well as capabilities. Our models and datasets are available at alignmentpretraining.ai

Related papers

• Synthetic Persona Pretraining: Alignment from Token Zero [20.4315235761248]
  We introduce Synthetic Persona Pretraining, which installs the desired assistant persona from token zero in pretraining.<n>By pretraining models up to 3B parameters on 500B tokens, we show that SPP improves constitution following and jailbreak robustness.<n>Overall, our results show that shaping values early is critical for alignment and establish pretraining-time persona interventions as an effective approach to do so.
  arXiv  Detail & Related papers  (2026-08-13T17:12:04Z)
• What Shapes Emergent Misalignment? Insights from Training Dynamics, Model Priors, and Data [20.335735139409245]
  Emergent misalignment (EM) is a phenomenon in which models generalize with narrow fine-tuning.<n>We study EM and its variability directly through the components of fine-tuning: training dynamics, model priors, and data.
  arXiv  Detail & Related papers  (2026-06-18T18:04:20Z)
• Conditional misalignment: common interventions can hide emergent misalignment behind contextual triggers [10.421731840102149]
  Finetuning a language model can lead to emergent misalignment (EM)<n>We study a set of interventions proposed to reduce EM.<n>If the evaluation prompts are tweaked to resemble the training context, the model displays EM.
  arXiv  Detail & Related papers  (2026-04-28T17:36:06Z)
• Alignment Imprint: Zero-Shot AI-Generated Text Detection via Provable Preference Discrepancy [51.887915969023965]
  We show that modern Large Language Models (LLMs) undergo alignment, leaving a measurable distributional imprint.<n>To mitigate the instability in high-entropy regions, we introduce Log-likelihood Alignment Preference Discrepancy (LAPD)<n>LAPD is a standardized information-weighted statistic based on alignment imprint.
  arXiv  Detail & Related papers  (2026-04-18T09:12:24Z)
• Alignment Faking - the Train -> Deploy Asymmetry: Through a Game-Theoretic Lens with Bayesian-Stackelberg Equilibria [16.451012162731047]
  Alignment faking is a form of strategic deception in AI.<n>Models selectively comply with training objectives when they infer that they are in training.<n>Our goal is to identify what causes alignment faking and when it occurs.
  arXiv  Detail & Related papers  (2025-11-22T06:30:51Z)
• A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques [63.10251271444959]
  Large language models are first pre-trained on trillions of tokens and then instruction-tuned or aligned to specific preferences. We conduct an in-depth investigation of the impact of popular choices for three crucial axes. Our setup spanning over 300 experiments reveals consistent trends and unexpected findings.
  arXiv  Detail & Related papers  (2024-06-07T12:25:51Z)
• Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback [70.32795295142648]
  Linear alignment is a novel algorithm that aligns language models with human preferences in one single inference step. Experiments on both general and personalized preference datasets demonstrate that linear alignment significantly enhances the performance and efficiency of LLM alignment.
  arXiv  Detail & Related papers  (2024-01-21T10:46:23Z)
• Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment [105.34140537748546]
  We propose an improved alignment approach named FIGA. Different from prior methods, we incorporate fine-grained quality signals that are derived by contrasting good and bad responses. Our approach has made two major contributions. Firstly, we curate a refined alignment dataset that pairs initial responses and the corresponding revised ones. Secondly, we devise a new loss function can leverage fine-grained quality signals to instruct the learning of LLMs for alignment.
  arXiv  Detail & Related papers  (2023-11-07T15:36:40Z)
• AI Alignment: A Comprehensive Survey [69.61425542486275]
  AI alignment aims to make AI systems behave in line with human intentions and values.<n>We identify four principles as the key objectives of AI alignment: Robustness, Interpretability, Controllability, and Ethicality.<n>We decompose current alignment research into two key components: forward alignment and backward alignment.
  arXiv  Detail & Related papers  (2023-10-30T15:52:15Z)
• Making Large Language Models Better Reasoners with Alignment [57.82176656663245]
  Reasoning is a cognitive process of using evidence to reach a sound conclusion. Recent studies reveal that fine-tuning LLMs on data with the chain of thought (COT) reasoning process can significantly enhance their reasoning capabilities. We introduce an textitAlignment Fine-Tuning (AFT) paradigm, which involves three steps.
  arXiv  Detail & Related papers  (2023-09-05T11:32:48Z)
• Unsupervised Dense Retrieval with Relevance-Aware Contrastive Pre-Training [81.3781338418574]
  We propose relevance-aware contrastive learning. We consistently improve the SOTA unsupervised Contriever model on the BEIR and open-domain QA retrieval benchmarks. Our method can not only beat BM25 after further pre-training on the target corpus but also serves as a good few-shot learner.
  arXiv  Detail & Related papers  (2023-06-05T18:20:27Z)
• Meet in the Middle: A New Pre-training Paradigm [41.52858444519968]
  Most language models (LMs) are trained and applied in an autoregressive left-to-right fashion. We propose a new pre-training paradigm with techniques that jointly improve the training data efficiency. We show the effectiveness of our pre-training paradigm with extensive experiments on both programming and natural language models.
  arXiv  Detail & Related papers  (2023-03-13T17:17:11Z)
• Pretraining Language Models with Human Preferences [21.724817280998696]
  Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM. Here, we explore alternative objectives for pretraining LMs in a way that also guides them to generate text aligned with human preferences.
  arXiv  Detail & Related papers  (2023-02-16T21:03:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.