Early Data Exposure Improves Robustness to Subsequent Fine-Tuning

• URL: http://arxiv.org/abs/2605.12705v1
• Date: Tue, 12 May 2026 20:08:00 GMT
• Title: Early Data Exposure Improves Robustness to Subsequent Fine-Tuning
• Abstract summary: We study how upstream training choices shape how robustly that capability is retained.<n>Early exposure - mixing post-training data into pretraining - consistently improves the frontier between retained upstream performance and downstream performance.<n>Our findings suggest that robustness to subsequent fine-tuning should be treated as a first-class objective of upstream training.
• Score: 26.380638571847815
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: How can we train models whose post-trained capabilities survive subsequent fine-tuning? Rather than focusing on downstream interventions to mitigate forgetting of upstream capabilities, we study how upstream training choices - that is, the manner in which a capability is acquired - shape how robustly that capability is retained. We investigate this question in a controlled three-stage language-model pipeline: pretraining, post-training to acquire a target capability, and downstream fine-tuning on a new objective. Across 135M and 1B models, two post-training domains, and two downstream fine-tuning tasks, we find that immediate post-training performance does not reliably predict retention after subsequent fine-tuning: training recipes that look equivalent immediately after post-training can retain the target capability very differently after subsequent fine-tuning. In particular, early exposure - mixing post-training data into pretraining - consistently improves the frontier between retained upstream performance and downstream performance. In compute-matched experiments, where the target data must be allocated between pretraining and post-training, we find that the optimum lies at neither extreme. Together with our other empirical and theoretical findings, this supports the view that post-training drives immediate specialization while early exposure improves robustness to later forgetting. Replay and dropout, typically used to mitigate forgetting as it occurs during fine-tuning, provide complementary gains to early exposure when applied during post-training. Our findings suggest that robustness to subsequent fine-tuning should be treated as a first-class objective of upstream training, addressed preventatively through choices like early exposure rather than reactively during fine-tuning itself.

Related papers

• Evaluating Federated Pre-Training: On the Reliability of Downstream Fine-Tuning and Intrinsic Evaluation [3.488204158798457]
  Federated pre-training offers a way to train foundation models on private or distributed data without centralizing the underlying datasets.<n>We study which evaluation protocol more reliably reflects federated pre-training quality.
  arXiv  Detail & Related papers  (2026-07-20T15:14:12Z)
• Midtraining Bridges Pretraining and Posttraining Distributions [73.84346031272473]
  "Midtraining" is a phase in which higher quality, often instruction-formatted data is mixed in at the end of pretraining.<n>We conduct the first systematic investigation of midtraining through experiments with language models pretrained from scratch.<n>We find that when compared after supervised fine-tuning, the effectiveness of midtraining is highest in the math and code domains.
  arXiv  Detail & Related papers  (2025-10-16T16:39:52Z)
• Front-Loading Reasoning: The Synergy between Pretraining and Post-Training Data [68.85234898614571]
  The prevailing paradigm for enhancing the reasoning abilities of LLMs revolves around post-training on high-quality, reasoning-intensive data.<n>While emerging literature suggests that reasoning data is increasingly incorporated also during the mid-training stage, the role of such data in pretraining remains unclear.<n>We conduct the first systematic study of how reasoning data-varying in scale, diversity, and quality-affects LLM performance when introduced at different stages of training.
  arXiv  Detail & Related papers  (2025-09-26T20:08:51Z)
• RLP: Reinforcement as a Pretraining Objective [103.45068938532923]
  We present an information-driven reinforcement pretraining objective that brings the core spirit of reinforcement learning -- exploration -- to the last phase of pretraining.<n>This training objective essentially encourages the model to think for itself before predicting what comes next, thus teaching an independent thinking behavior earlier in the pretraining.<n> Specifically, RLP reframes reinforcement learning for reasoning as a pretraining objective on ordinary text, bridging the gap between next-token prediction and the emergence of useful chain-of-thought reasoning.
  arXiv  Detail & Related papers  (2025-09-26T17:53:54Z)
• A Comparative Study of Pre-training and Self-training [0.40964539027092917]
  We propose an ensemble method to empirical study all feasible training paradigms combining pre-training, self-training, and fine-tuning. We conduct experiments on six datasets, four data augmentation, and imbalanced data for sentiment analysis and natural language inference tasks. Our findings confirm that the pre-training and fine-tuning paradigm yields the best overall performances.
  arXiv  Detail & Related papers  (2024-09-04T14:30:13Z)
• Understanding and Mitigating the Label Noise in Pre-training on Downstream Tasks [91.15120211190519]
  This paper aims to understand the nature of noise in pre-training datasets and to mitigate its impact on downstream tasks. We propose a light-weight black-box tuning method (NMTune) to affine the feature space to mitigate the malignant effect of noise.
  arXiv  Detail & Related papers  (2023-09-29T06:18:15Z)
• Examining the Effect of Pre-training on Time Series Classification [21.38211396933795]
  This study investigates the impact of pre-training followed by fine-tuning on the fine-tuning process. We conducted a thorough examination of 150 classification datasets. We find that pre-training can only help improve the optimization process for models that fit the data poorly. Adding more pre-training data does not improve generalization, but it can strengthen the advantage of pre-training on the original data volume.
  arXiv  Detail & Related papers  (2023-09-11T06:26:57Z)
• Self-Distillation for Further Pre-training of Transformers [83.84227016847096]
  We propose self-distillation as a regularization for a further pre-training stage. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks.
  arXiv  Detail & Related papers  (2022-09-30T02:25:12Z)
• On Transfer of Adversarial Robustness from Pretraining to Downstream Tasks [1.8900691517352295]
  We show that the robustness of a linear predictor on downstream tasks can be constrained by the robustness of its underlying representation. Our results offer an initial step towards characterizing the requirements of the representation function for reliable post-adaptation performance.
  arXiv  Detail & Related papers  (2022-08-07T23:00:40Z)
• Improved Fine-tuning by Leveraging Pre-training Data: Theory and Practice [52.11183787786718]
  Fine-tuning a pre-trained model on the target data is widely used in many deep learning applications. Recent studies have empirically shown that training from scratch has the final performance that is no worse than this pre-training strategy. We propose a novel selection strategy to select a subset from pre-training data to help improve the generalization on the target task.
  arXiv  Detail & Related papers  (2021-11-24T06:18:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.