Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models

• URL: http://arxiv.org/abs/2407.07263v1
• Date: Tue, 9 Jul 2024 22:37:59 GMT
• Title: Reuse, Don't Retrain: A Recipe for Continued Pretraining of Language Models
• Authors: Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, Bryan Catanzaro,
• Abstract summary: We show how to design efficacious data distributions and learning rate schedules for continued pretraining of language models. We show an improvement of 9% in average model accuracy compared to the baseline of continued training on the pretraining set.
• Score: 29.367678364485794
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As language models have scaled both their number of parameters and pretraining dataset sizes, the computational cost for pretraining has become intractable except for the most well-resourced teams. This increasing cost makes it ever more important to be able to reuse a model after it has completed pretraining; allowing for a model's abilities to further improve without needing to train from scratch. In this work, we detail a set of guidelines that cover how to design efficacious data distributions and learning rate schedules for continued pretraining of language models. When applying these findings within a continued pretraining run on top of a well-trained 15B parameter model, we show an improvement of 9\% in average model accuracy compared to the baseline of continued training on the pretraining set. The resulting recipe provides a practical starting point with which to begin developing language models through reuse rather than retraining.

Related papers

• Scaling Smart: Accelerating Large Language Model Pre-training with Small Model Initialization [22.90653167145603]
  We introduce HyperCloning, a method that can expand the parameters of a pre-trained language model to those of a larger model with increased hidden dimensions. As a result, the larger model already inherits the predictive power and accuracy of the smaller model before the training starts.
  arXiv  Detail & Related papers  (2024-09-19T16:50:26Z)
• Amuro & Char: Analyzing the Relationship between Pre-Training and Fine-Tuning of Large Language Models [17.288865972774587]
  We investigate the relationship between pre-training and fine-tuning by fine-tuning multiple intermediate pre-trained model checkpoints. Our results on 18 datasets suggest that pre-training improves the model in a latent way that unveils after fine-tuning.
  arXiv  Detail & Related papers  (2024-08-13T06:28:43Z)
• Self-Distillation for Further Pre-training of Transformers [83.84227016847096]
  We propose self-distillation as a regularization for a further pre-training stage. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks.
  arXiv  Detail & Related papers  (2022-09-30T02:25:12Z)
• Continual Pre-Training Mitigates Forgetting in Language and Vision [43.80547864450793]
  We show that continually pre-trained models are robust against catastrophic forgetting. We provide empirical evidence supporting the fact that self-supervised pre-training is more effective in retaining previous knowledge than supervised protocols.
  arXiv  Detail & Related papers  (2022-05-19T07:27:12Z)
• Arithmetic-Based Pretraining -- Improving Numeracy of Pretrained Language Models [67.48894919842576]
  State-of-the-art pretrained language models tend to perform below their capabilities when applied out-of-the-box on tasks that require numeracy. We propose a new extended pretraining approach called Arithmetic-Based Pretraining that jointly addresses both in one extended pretraining step. Our experiments show the effectiveness of Arithmetic-Based Pretraining in three different tasks that require improved numeracy.
  arXiv  Detail & Related papers  (2022-05-13T16:10:13Z)
• bert2BERT: Towards Reusable Pretrained Language Models [51.078081486422896]
  We propose bert2BERT, which can effectively transfer the knowledge of an existing smaller pre-trained model to a large model. bert2BERT saves about 45% and 47% computational cost of pre-training BERT_BASE and GPT_BASE by reusing the models of almost their half sizes.
  arXiv  Detail & Related papers  (2021-10-14T04:05:25Z)
• Self-Supervised Pretraining Improves Self-Supervised Pretraining [83.1423204498361]
  Self-supervised pretraining requires expensive and lengthy computation, large amounts of data, and is sensitive to data augmentation. This paper explores Hierarchical PreTraining (HPT), which decreases convergence time and improves accuracy by initializing the pretraining process with an existing pretrained model. We show HPT converges up to 80x faster, improves accuracy across tasks, and improves the robustness of the self-supervised pretraining process to changes in the image augmentation policy or amount of pretraining data.
  arXiv  Detail & Related papers  (2021-03-23T17:37:51Z)
• Pre-Training a Language Model Without Human Language [74.11825654535895]
  We study how the intrinsic nature of pre-training data contributes to the fine-tuned downstream performance. We find that models pre-trained on unstructured data beat those trained directly from scratch on downstream tasks. To our great astonishment, we uncover that pre-training on certain non-human language data gives GLUE performance close to performance pre-trained on another non-English language.
  arXiv  Detail & Related papers  (2020-12-22T13:38:06Z)
• On the importance of pre-training data volume for compact language models [0.7691755449724638]
  We study the impact of pre-training data volume on compact language models. We observe that well-performing models are obtained with as little as 100 MB of text.
  arXiv  Detail & Related papers  (2020-10-08T07:40:21Z)
• Exploring Fine-tuning Techniques for Pre-trained Cross-lingual Models via Continual Learning [74.25168207651376]
  Fine-tuning pre-trained language models to downstream cross-lingual tasks has shown promising results. We leverage continual learning to preserve the cross-lingual ability of the pre-trained model when we fine-tune it to downstream tasks. Our methods achieve better performance than other fine-tuning baselines on the zero-shot cross-lingual part-of-speech tagging and named entity recognition tasks.
  arXiv  Detail & Related papers  (2020-04-29T14:07:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.