Domain-Adaptive Continued Pre-Training of Small Language Models

• URL: http://arxiv.org/abs/2504.09687v1
• Date: Sun, 13 Apr 2025 18:40:32 GMT
• Title: Domain-Adaptive Continued Pre-Training of Small Language Models
• Authors: Salman Faroz,
• Abstract summary: Continued pre-training of small language models offers a promising path for domain adaptation with limited computational resources.<n>I've investigated this approach within educational domains, evaluating it as a resource-efficient alternative to training models from scratch.<n>My approach includes comprehensive data preprocessing, memory-optimized training configurations, and benchmark-based evaluation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continued pre-training of small language models offers a promising path for domain adaptation with limited computational resources. I've investigated this approach within educational domains, evaluating it as a resource-efficient alternative to training models from scratch. Using a 125M parameter model, I demonstrate significant performance improvements through incremental training on 400 million tokens, followed by further training to reach 1 billion tokens. My approach includes comprehensive data preprocessing, memory-optimized training configurations, and benchmark-based evaluation. Results show notable gains in knowledge-intensive tasks (MMLU +8.1%) and contextual understanding (HellaSwag +7.6%), while revealing educational domain specialization trade-offs. I analyze token efficiency, catastrophic forgetting mitigation strategies, and scaling patterns. My findings suggest that thoughtful preprocessing and training methodologies enable meaningful improvements in language model capabilities even with constrained computational resources, opening pathways for domain-specific adaptation of smaller language models.

Related papers

• Training Dynamics of a 1.7B LLaMa Model: A Data-Efficient Approach [10.39475177812483]
  We share insights gained from training DMaS-LLaMa-Lite on approximately 20 billion tokens of data.<n>We chronicle the full training trajectory, documenting how evolving validation loss levels and downstream benchmarks reflect transitions from incoherent text to fluent, contextually grounded output.<n>By detailing these experiences and offering training logs, checkpoints, and sample outputs, we aim to guide future researchers and practitioners in refining their pretraining strategies.
  arXiv  Detail & Related papers  (2024-12-17T21:15:52Z)
• What Do Learning Dynamics Reveal About Generalization in LLM Reasoning? [83.83230167222852]
  We find that a model's generalization behavior can be effectively characterized by a training metric we call pre-memorization train accuracy. By connecting a model's learning behavior to its generalization, pre-memorization train accuracy can guide targeted improvements to training strategies.
  arXiv  Detail & Related papers  (2024-11-12T09:52:40Z)
• Accelerating Large Language Model Pretraining via LFR Pedagogy: Learn, Focus, and Review [50.78587571704713]
  Learn-Focus-Review (LFR) is a dynamic training approach that adapts to the model's learning progress. LFR tracks the model's learning performance across data blocks (sequences of tokens) and prioritizes revisiting challenging regions of the dataset. Compared to baseline models trained on the full datasets, LFR consistently achieved lower perplexity and higher accuracy.
  arXiv  Detail & Related papers  (2024-09-10T00:59:18Z)
• IGOT: Information Gain Optimized Tokenizer on Domain Adaptive Pretraining [2.009700777745832]
  Pretrained Large Language Models (LLM) have demonstrated strong capabilities in various fields of natural language generation. When using generative AI to process downstream tasks, a common approach is to add new knowledge through continued training or fine-tuning. In this article, we proposed Information Gain Optimized Tokenizer (IGOT) which analyzes the special token set of downstream tasks, constructs a new subset using $phi$ with the special token and its information gain. We explored the many positive effects of this method's customized tokenizer on domain-adaptive pretraining and verified this method can perform better than the
  arXiv  Detail & Related papers  (2024-05-16T07:25:10Z)
• Irreducible Curriculum for Language Model Pretraining [46.895234111411426]
  We propose irreducible curriculum as a curriculum learning algorithm for language model pretraining. Our experiments on the RedPajama-1B dataset demonstrate a consistent improvement on validation perplexity across all 7 domains.
  arXiv  Detail & Related papers  (2023-10-23T22:41:33Z)
• Improving Pre-trained Language Model Fine-tuning with Noise Stability Regularization [94.4409074435894]
  We propose a novel and effective fine-tuning framework, named Layerwise Noise Stability Regularization (LNSR) Specifically, we propose to inject the standard Gaussian noise and regularize hidden representations of the fine-tuned model. We demonstrate the advantages of the proposed method over other state-of-the-art algorithms including L2-SP, Mixout and SMART.
  arXiv  Detail & Related papers  (2022-06-12T04:42:49Z)
• EMS: Efficient and Effective Massively Multilingual Sentence Embedding Learning [38.928786416891924]
  We introduce efficient and effective massively multilingual sentence embedding (EMS) using cross-lingual token-level reconstruction (XTR) and sentence-level contrastive learning as training objectives. Compared with related studies, the proposed model can be efficiently trained using significantly fewer parallel sentences and GPU computation resources. We release the codes for model training and the EMS pre-trained sentence embedding model, which supports 62 languages.
  arXiv  Detail & Related papers  (2022-05-31T12:29:25Z)
• Efficient Domain Adaptation of Language Models via Adaptive Tokenization [5.058301279065432]
  We show that domain-specific subword sequences can be efficiently determined directly from divergences in the conditional token distributions of the base and domain-specific corpora. Our approach produces smaller models and less training and inference time than other approaches using tokenizer augmentation.
  arXiv  Detail & Related papers  (2021-09-15T17:51:27Z)
• Efficient Nearest Neighbor Language Models [114.40866461741795]
  Non-parametric neural language models (NLMs) learn predictive distributions of text utilizing an external datastore. We show how to achieve up to a 6x speed-up in inference speed while retaining comparable performance.
  arXiv  Detail & Related papers  (2021-09-09T12:32:28Z)
• ALLWAS: Active Learning on Language models in WASserstein space [13.35098213857704]
  In several domains, such as medicine, the scarcity of labeled training data is a common issue. Active learning may prove helpful in these cases to boost the performance with a limited label budget. We propose a novel method using sampling techniques based on submodular optimization and optimal transport for active learning in language models.
  arXiv  Detail & Related papers  (2021-09-03T18:11:07Z)
• Selecting Informative Contexts Improves Language Model Finetuning [66.26521454263343]
  We present a general fine-tuning method that we call information gain filtration. During fine-tuning, a secondary learner selects informative examples and skips uninformative ones. We show that our method has consistent improvement across datasets, fine-tuning tasks, and language model architectures.
  arXiv  Detail & Related papers  (2020-05-01T02:01:18Z)
• Pre-training Text Representations as Meta Learning [113.3361289756749]
  We introduce a learning algorithm which directly optimize model's ability to learn text representations for effective learning of downstream tasks. We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps.
  arXiv  Detail & Related papers  (2020-04-12T09:05:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.