When is Warmstarting Effective for Scaling Language Models?

• URL: http://arxiv.org/abs/2605.13405v1
• Date: Wed, 13 May 2026 12:00:11 GMT
• Title: When is Warmstarting Effective for Scaling Language Models?
• Abstract summary: We show that preserving the base model's initial post-growth performance is not necessary for strong final performance.<n>We empirically identify an upper bound on the growth factor $g$ beyond which training from scratch is more efficient.
• Score: 48.713404953553976
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Model growth from a given checkpoint aims to accelerate training of a larger model, offering potential resource savings. Despite recent interest, warmstarting has seen limited practical adoption in large-scale training. We attribute this to two underexplored factors: (1) an overemphasis on preserving the smaller model's performance at initialization, which constrains operator design for new architectures, and (2) insufficient analysis of how growth interacts with hyperparameters and scaling behavior, compounded by inconsistent growth factors across the literature. We show that preserving the base model's initial post-growth performance is not necessary for strong final performance, and that simple, architecture-agnostic growth strategies can outperform more complex warmstarting operators. Crucially, we empirically identify an upper bound on the growth factor $g$ beyond which training from scratch is more efficient. We observe this across multiple ablation setups. Notably, this limit is also present, but unreported, in prior published results. Across our experiments on dense MLPs and dense language models, we find that a $2\times$ growth factor is the most reliable in yielding convergence speedups, with gains most pronounced under 20 tokens/parameter budgets and diminishing as budget increases. We fit scaling laws over these observations to provide predictive guidance for practitioners deciding when and how much to grow. Together, our analysis provides practical guidelines and empirical limits for model growth.

Related papers

• Scaling Laws of Global Weather Models [57.27583619011988]
  We investigate the relationship between model performance (validation loss) and three key factors: model size, dataset size, and compute budget.<n>Across a range of models, we find that Aurora exhibits the strongest data-scaling behavior.<n>Our compute-optimal analysis indicates that, under fixed compute budgets, allocating resources to longer training durations yields greater performance gains than increasing model size.
  arXiv  Detail & Related papers  (2026-02-26T12:57:38Z)
• Recycling Pretrained Checkpoints: Orthogonal Growth of Mixture-of-Experts for Efficient Large Language Model Pre-Training [70.60554423630803]
  We propose to recycle pretrained checkpoints by expanding their parameter counts and continuing training.<n>We scale our approach to models with 70B parameters and over 1T training tokens, achieving 10.66% accuracy gain over training from scratch.
  arXiv  Detail & Related papers  (2025-10-09T09:45:45Z)
• Mitigating Catastrophic Forgetting in Continual Learning through Model Growth [3.027322177899458]
  Catastrophic forgetting is a significant challenge in continual learning, in which a model loses prior knowledge when it is fine-tuned on new tasks.<n>This problem is particularly critical for large language models undergoing continual learning, as retaining performance across diverse domains is important for their general utility.<n>We evaluate whether growth-based pretraining can retain previously learned capabilities more effectively across a sequence of fine-tuning tasks involving domain knowledge, reasoning, reading comprehension, and bias.
  arXiv  Detail & Related papers  (2025-09-01T07:51:31Z)
• Scaling over Scaling: Exploring Test-Time Scaling Plateau in Large Reasoning Models [7.2703757624760526]
  Large reasoning models (LRMs) have exhibited the capacity of enhancing reasoning performance via internal test-time scaling.<n>As we push these scaling boundaries, understanding the practical limits and achieving optimal resource allocation becomes a critical challenge.<n>In this paper, we investigate the scaling plateau of test-time scaling and introduce the Test-Time Scaling Performance Model (TTSPM)
  arXiv  Detail & Related papers  (2025-05-26T20:58:45Z)
• LESA: Learnable LLM Layer Scaling-Up [57.0510934286449]
  Training Large Language Models (LLMs) from scratch requires immense computational resources, making it prohibitively expensive.<n>Model scaling-up offers a promising solution by leveraging the parameters of smaller models to create larger ones.<n>We propose textbfLESA, a novel learnable method for depth scaling-up.
  arXiv  Detail & Related papers  (2025-02-19T14:58:48Z)
• Landscape-Aware Growing: The Power of a Little LAG [49.897766925371485]
  We study the question of how to select the best growing strategy from a given pool of growing strategies. We present an alternative perspective based on early training dynamics, which we call "landscape-aware growing (LAG)"
  arXiv  Detail & Related papers  (2024-06-04T16:38:57Z)
• PanGu-$\pi$: Enhancing Language Model Architectures via Nonlinearity Compensation [97.78045712375047]
  We present a new efficient model architecture for large language models (LLMs) We show that PanGu-$pi$-7B can achieve a comparable performance to that of benchmarks with about 10% inference speed-up. In addition, we have deployed PanGu-$pi$-7B in the high-value domains of finance and law, developing an LLM named YunShan for practical application.
  arXiv  Detail & Related papers  (2023-12-27T11:49:24Z)
• Predicting Emergent Abilities with Infinite Resolution Evaluation [85.89911520190711]
  We introduce PassUntil, an evaluation strategy with theoretically infinite resolution, through massive sampling in the decoding phase. We predict the performance of the 2.4B model on code generation with merely 0.05% deviation before training starts. We identify a kind of accelerated emergence whose scaling curve cannot be fitted by standard scaling law function.
  arXiv  Detail & Related papers  (2023-10-05T02:35:00Z)
• Masked Structural Growth for 2x Faster Language Model Pre-training [18.276784451675603]
  We focus on speeding up pre-training by progressively growing from a small Transformer structure to a large one. In terms of growth schedule, the impact of each single dimension on a schedule's efficiency is under-explored by existing work. We propose Masked Structural Growth (MSG), including (i) growth schedules involving all possible dimensions and (ii) strictly function-preserving growth operators.
  arXiv  Detail & Related papers  (2023-05-04T14:28:39Z)
• On the Transformer Growth for Progressive BERT Training [37.57617077192438]
  We find that similar to network architecture search, Transformer growth also favors compound scaling. In light of our analyses, the proposed method speeds up BERT pre-training by 73.6% and 82.2% for the base and large models respectively.
  arXiv  Detail & Related papers  (2020-10-23T17:44:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.