The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data

• URL: http://arxiv.org/abs/2603.16177v2
• Date: Sat, 21 Mar 2026 01:17:39 GMT
• Title: The Finetuner's Fallacy: When to Pretrain with Your Finetuning Data
• Abstract summary: We study a simple strategy, specialized pretraining (SPT), where a small domain dataset is repeated starting from pretraining as a fraction of the total tokens.<n>In our experiments, SPT reduces the pretraining tokens needed to reach a given domain performance by up to 1.75x.<n>Our observations reveal the finetuner's fallacy: while finetuning may appear to be the cheapest path to domain adaptation, introducing specialized domain data during pretraining stretches its utility.
• Score: 55.87500250831868
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-world model deployments demand strong performance on narrow domains where data is often scarce. Typically, practitioners finetune models to specialize them, but this risks overfitting to the domain and forgetting general knowledge. We study a simple strategy, specialized pretraining (SPT), where a small domain dataset, typically reserved for finetuning, is repeated starting from pretraining as a fraction of the total tokens. Across three specialized domains (ChemPile, MusicPile, and ProofPile), SPT improves domain performance and preserves general capabilities after finetuning compared to standard pretraining. In our experiments, SPT reduces the pretraining tokens needed to reach a given domain performance by up to 1.75x. These gains grow when the target domain is underrepresented in the pretraining corpus: on domains far from web text, a 1B SPT model outperforms a 3B standard pretrained model. Beyond these empirical gains, we derive overfitting scaling laws to guide practitioners in selecting the optimal domain-data repetition for a given pretraining compute budget. Our observations reveal the finetuner's fallacy: while finetuning may appear to be the cheapest path to domain adaptation, introducing specialized domain data during pretraining stretches its utility. SPT yields better specialized domain performance (via reduced overfitting across repeated exposures) and better general domain performance (via reduced forgetting during finetuning), ultimately achieving stronger results with fewer parameters and less total compute when amortized over inference. To get the most out of domain data, incorporate it as early in training as possible.

Related papers

• Scaling Domain Data Repetition in LLM Pretraining [18.481235801370826]
  High-quality domain data is much harder to scale than general web data.<n>As model size and the training-token budget increase, its fraction in the training mixture tends to decrease.<n>We find that repetition counts tuned on smaller proxy models with the same (mathrmTPP) can provide a practical estimate for larger models.
  arXiv  Detail & Related papers  (2026-08-14T08:27:15Z)
• Is Large-Scale Pretraining the Secret to Good Domain Generalization? [69.80606575323691]
  Multi-Source Domain Generalization (DG) is the task of training on multiple source domains and achieving high classification performance on unseen target domains.<n>Recent methods combine robust features from web-scale pretrained backbones with new features learned from source data, and this has dramatically improved benchmark results.<n>We show that all evaluated DG methods struggle on DomainBed-OOP, while recent methods excel on DomainBed-IP.
  arXiv  Detail & Related papers  (2024-12-03T21:43:11Z)
• IGOT: Information Gain Optimized Tokenizer on Domain Adaptive Pretraining [2.009700777745832]
  Pretrained Large Language Models (LLM) have demonstrated strong capabilities in various fields of natural language generation. When using generative AI to process downstream tasks, a common approach is to add new knowledge through continued training or fine-tuning. In this article, we proposed Information Gain Optimized Tokenizer (IGOT) which analyzes the special token set of downstream tasks, constructs a new subset using $phi$ with the special token and its information gain. We explored the many positive effects of this method's customized tokenizer on domain-adaptive pretraining and verified this method can perform better than the
  arXiv  Detail & Related papers  (2024-05-16T07:25:10Z)
• AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification [53.84663977283108]
  Few-shot sentence classification based on pre-trained Sentences (SEs) is efficient, robust, and effective. We propose AdaSent, which decouples SEPT from DAPT by training a SEPT adapter on the base PLM. AdaSent matches or surpasses the performance of full SEPT on DAPT-ed PLM, while substantially reducing the training costs.
  arXiv  Detail & Related papers  (2023-11-01T10:00:15Z)
• Regex-augmented Domain Transfer Topic Classification based on a Pre-trained Language Model: An application in Financial Domain [42.5087655999509]
  We discuss the use of regular expression patterns employed as features for domain knowledge during the process of fine tuning. Our experiments on real scenario production data show that this method of fine tuning improves the downstream text classification tasks.
  arXiv  Detail & Related papers  (2023-05-23T03:26:32Z)
• Forget Less, Count Better: A Domain-Incremental Self-Distillation Learning Benchmark for Lifelong Crowd Counting [51.44987756859706]
  Off-the-shelf methods have some drawbacks to handle multiple domains. Lifelong Crowd Counting aims at alleviating the catastrophic forgetting and improving the generalization ability.
  arXiv  Detail & Related papers  (2022-05-06T15:37:56Z)
• Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training [67.71228426496013]
  We show that using target domain data during pre-training leads to large performance improvements across a variety of setups. We find that pre-training on multiple domains improves performance generalization on domains not seen during training.
  arXiv  Detail & Related papers  (2021-04-02T12:53:15Z)
• Batch Normalization Embeddings for Deep Domain Generalization [50.51405390150066]
  Domain generalization aims at training machine learning models to perform robustly across different and unseen domains. We show a significant increase in classification accuracy over current state-of-the-art techniques on popular domain generalization benchmarks.
  arXiv  Detail & Related papers  (2020-11-25T12:02:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.