Synthetic Persona Pretraining: Alignment from Token Zero

• URL: http://arxiv.org/abs/2608.13482v1
• Date: Thu, 13 Aug 2026 17:12:04 GMT
• Title: Synthetic Persona Pretraining: Alignment from Token Zero
• Abstract summary: We introduce Synthetic Persona Pretraining, which installs the desired assistant persona from token zero in pretraining.<n>By pretraining models up to 3B parameters on 500B tokens, we show that SPP improves constitution following and jailbreak robustness.<n>Overall, our results show that shaping values early is critical for alignment and establish pretraining-time persona interventions as an effective approach to do so.
• Score: 20.4315235761248
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As language-model-based AI is increasingly deployed in autonomous settings, aligning its goals and values with those of humans becomes critical. Today, alignment, and the assistant identity itself, are typically introduced only after pretraining, once behavioral priors are already established. This can make values a thin overlay, rather than deeply rooted, and facilitate subsequent misalignment. Pursuing a different paradigm, we introduce Synthetic Persona Pretraining (SPP), which installs the desired assistant persona from token zero in pretraining. First, we annotate pretraining documents with value-aligned first-person reflections derived from a normative value constitution. Second, we pretrain via the standard cross-entropy loss on standard pretraining documents as well as their reflections, which installs the desired persona among a multitude of other personas. Finally, we post-train on user-assistant dialogue data, which binds this desired persona to the assistant identity, a process we call persona binding. By pretraining models up to 3B parameters on 500B tokens, we show that SPP improves constitution following and jailbreak robustness, and reduces the misalignment rate in out-of-distribution moral dilemmas, while preserving capabilities. Early intervention matters: compared with alignment from token zero, introducing SPP only at the end of pretraining yields weaker constitution adherence, does not shift value priorities, and leads to less aligned choices in dilemmas. This advantage depends on persona binding and, importantly, increases with pretraining budget. Overall, our results show that shaping values early is critical for alignment and establish pretraining-time persona interventions as an effective approach to do so.

Related papers

• Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation [3.488204158798457]
  Federated pre-training offers a way to train foundation models on private or distributed data without centralizing the underlying datasets.<n>We study which evaluation protocol more reliably reflects federated pre-training quality.
  arXiv  Detail & Related papers  (2026-07-20T15:14:12Z)
• APAO: Adaptive Prefix-Aware Optimization for Generative Recommendation [26.371939617653084]
  Generative recommendation is an autoregressive generation process, predicting discrete tokens of the next item conditioned on user interaction histories.<n>Existing generative recommendation models are typically trained with token-level likelihood objectives, such as cross-entropy loss.<n>This leads to a training-inference inconsistency: standard training assumes ground-truth history is always available, ignoring the fact that beam search prunes low-probability branches during inference.
  arXiv  Detail & Related papers  (2026-03-03T08:29:15Z)
• Alignment Pretraining: AI Discourse Causes Self-Fulfilling (Mis)alignment [2.5029856851310552]
  We find that discussion of AI contributes to misalignment.<n>We consider this evidence of self-fulfilling alignment.<n>Our findings establish the study of how pretraining data shapes alignment priors, or alignment pretraining, as a complement to post-training.
  arXiv  Detail & Related papers  (2026-01-15T07:59:31Z)
• RLP: Reinforcement as a Pretraining Objective [103.45068938532923]
  We present an information-driven reinforcement pretraining objective that brings the core spirit of reinforcement learning -- exploration -- to the last phase of pretraining.<n>This training objective essentially encourages the model to think for itself before predicting what comes next, thus teaching an independent thinking behavior earlier in the pretraining.<n> Specifically, RLP reframes reinforcement learning for reasoning as a pretraining objective on ordinary text, bridging the gap between next-token prediction and the emergence of useful chain-of-thought reasoning.
  arXiv  Detail & Related papers  (2025-09-26T17:53:54Z)
• Synthetic bootstrapped pretraining [52.92577542049469]
  We introduce Synthetic Bootstrapped Pretraining (SBP), a language model (LM) pretraining procedure.<n>SBP first learns a model of relations between documents from the pretraining dataset and then leverages it to synthesize a vast new corpus for joint training.<n>We find SBP consistently improves upon a strong repetition baseline and delivers a significant fraction of performance improvement attainable by an oracle upper bound.
  arXiv  Detail & Related papers  (2025-09-17T22:28:27Z)
• Why Is Public Pretraining Necessary for Private Model Training? [50.054565310457306]
  We show that pretraining on publicly available data leads to distinct gains over nonprivate settings. We argue that the tradeoff may be a deeper loss model that requires an algorithm to go through two phases. Guided by intuition, we provide theoretical constructions that provably demonstrate the separation between private with and without public pretraining.
  arXiv  Detail & Related papers  (2023-02-19T05:32:20Z)
• Pretraining Language Models with Human Preferences [21.724817280998696]
  Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM. Here, we explore alternative objectives for pretraining LMs in a way that also guides them to generate text aligned with human preferences.
  arXiv  Detail & Related papers  (2023-02-16T21:03:33Z)
• Adversarial Pretraining of Self-Supervised Deep Networks: Past, Present and Future [132.34745793391303]
  We review adversarial pretraining of self-supervised deep networks including both convolutional neural networks and vision transformers. To incorporate adversaries into pretraining models on either input or feature level, we find that existing approaches are largely categorized into two groups.
  arXiv  Detail & Related papers  (2022-10-23T13:14:06Z)
• Revisiting Membership Inference Under Realistic Assumptions [87.13552321332988]
  We study membership inference in settings where some of the assumptions typically used in previous research are relaxed. This setting is more realistic than the balanced prior setting typically considered by researchers. We develop a new inference attack based on the intuition that inputs corresponding to training set members will be near a local minimum in the loss function.
  arXiv  Detail & Related papers  (2020-05-21T20:17:42Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.