Spike No More: Stabilizing the Pre-training of Large Language Models

• URL: http://arxiv.org/abs/2312.16903v4
• Date: Fri, 25 Jul 2025 05:09:17 GMT
• Title: Spike No More: Stabilizing the Pre-training of Large Language Models
• Abstract summary: Loss spikes often occur during pre-training of large language models.<n>We explore factors to keep the gradient norm small through an analysis of the spectral norms of the Jacobian matrices for the sub-layers.<n>Our findings suggest that stabilizing the pre-training process requires two conditions: small sub-layers and large shortcut.
• Score: 38.34737204307412
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Loss spikes often occur during pre-training of large language models. The spikes degrade the performance of large language models and sometimes ruin the pre-training. Since the pre-training needs a vast computational budget, we should avoid such spikes. Based on the assumption that the loss spike is caused by the sudden growth of the gradient norm, we explore factors to keep the gradient norm small through an analysis of the spectral norms of the Jacobian matrices for the sub-layers. Our findings suggest that stabilizing the pre-training process requires two conditions: small sub-layers and large shortcut. We conduct various experiments to empirically verify our theoretical analyses. Experimental results demonstrate that methods satisfying the conditions effectively prevent loss spikes during pre-training.

Related papers

• Nonsense Helps: Prompt Space Perturbation Broadens Reasoning Exploration [14.622332371428056]
  We propose Lorem Perturbation for Exploration (LoPE) to break the exploration bottleneck.<n>LoPE prepends sequences assembled from Lorem Ipsum vocabulary (a pseudo-Latin placeholder text) to the prompts before resampling.<n>Our results establish LoPE as a strong baseline for broadening exploration in Large Language Models.
  arXiv  Detail & Related papers  (2026-05-07T01:18:06Z)
• A Mechanism Study of Delayed Loss Spikes in Batch-Normalized Linear Models [28.92895415506565]
  Delayed loss spikes have been reported in neural-network training.<n>Existing theory mainly explains earlier non-monotone behavior caused by overly large fixed learning rates.<n>We study one hypothesis: normalization can postpone instability by gradually increasing the effective learning rate.
  arXiv  Detail & Related papers  (2026-04-18T03:42:05Z)
• To Grok Grokking: Provable Grokking in Ridge Regression [24.785366757570202]
  We study grokking, the onset of generalization long after overfitting, in a classical ridge regression setting.<n>We show, both theoretically and empirically, that grokking can be amplified or eliminated in a principled manner.<n>Our results suggest grokking is not an inherent failure mode of deep learning, but rather a consequence of specific training conditions.
  arXiv  Detail & Related papers  (2026-01-27T16:52:04Z)
• Regularity as Structural Amplifier, Not Trap: A Causal and Archetype-Based Analysis of Dropout in a Constrained Engineering Curriculum [0.0]
  Engineering programmes, particularly in Latin America, are often governed by rigid curricula and strict regularity rules.<n>This study tests that causal hypothesis using the CAPIRE framework.
  arXiv  Detail & Related papers  (2025-11-24T10:53:41Z)
• SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization [62.958457694151384]
  We introduce preference optimization into precipitation nowcasting for the first time, motivated by the success of reinforcement learning from human feedback in large language models.<n>In the first stage, the framework focuses on reducing FAR, training the model to effectively suppress false alarms.
  arXiv  Detail & Related papers  (2025-10-22T16:11:22Z)
• The Coverage Principle: How Pre-Training Enables Post-Training [70.25788947586297]
  We study how pre-training shapes the success of the final model.<n>We uncover a mechanism that explains the power of coverage in predicting downstream performance.
  arXiv  Detail & Related papers  (2025-10-16T17:53:50Z)
• Temporal horizons in forecasting: a performance-learnability trade-off [1.564538938595125]
  Too short a horizon may miss long-term trends, while too long a horizon can impede convergence due to accumulating prediction errors.<n>We prove that for chaotic systems, the loss landscape's roughness grows exponentially with the training horizon.<n>We also show that models trained on long horizons generalize well to short-term forecasts, whereas those trained on short horizons suffer exponentially (resp. linearly) worse long-term predictions in chaotic (resp. periodic) systems.
  arXiv  Detail & Related papers  (2025-06-04T12:34:22Z)
• Overtrained Language Models Are Harder to Fine-Tune [64.44743256512237]
  Large language models are pre-trained on ever-growing token budgets. We show that extended pre-training can make models harder to fine-tune, leading to degraded final performance.
  arXiv  Detail & Related papers  (2025-03-24T23:11:56Z)
• Adaptive scheduling for adaptive sampling in POS taggers construction [0.27624021966289597]
  We introduce an adaptive scheduling for adaptive sampling as a novel way of machine learning in the construction of part-of-speech taggers. We analyze the shape of the learning curve geometrically in conjunction with a functional model to increase or decrease it at any time. We also improve the robustness of sampling by paying greater attention to those regions of the training data base subject to a temporary inflation in performance.
  arXiv  Detail & Related papers  (2024-02-04T15:02:17Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Small-scale proxies for large-scale Transformer training instabilities [69.36381318171338]
  We seek ways to reproduce and study training stability and instability at smaller scales. By measuring the relationship between learning rate and loss across scales, we show that these instabilities also appear in small models when training at high learning rates. We study methods such as warm-up, weight decay, and the $mu$Param to train small models that achieve similar losses across orders of magnitude of learning rate variation.
  arXiv  Detail & Related papers  (2023-09-25T17:48:51Z)
• Examining the Effect of Pre-training on Time Series Classification [21.38211396933795]
  This study investigates the impact of pre-training followed by fine-tuning on the fine-tuning process. We conducted a thorough examination of 150 classification datasets. We find that pre-training can only help improve the optimization process for models that fit the data poorly. Adding more pre-training data does not improve generalization, but it can strengthen the advantage of pre-training on the original data volume.
  arXiv  Detail & Related papers  (2023-09-11T06:26:57Z)
• Dropout Reduces Underfitting [85.61466286688385]
  In this study, we demonstrate that dropout can also mitigate underfitting when used at the start of training. We find dropout reduces the directional variance of gradients across mini-batches and helps align the mini-batch gradients with the entire dataset's gradient. Our findings lead us to a solution for improving performance in underfitting models - early dropout: dropout is applied only during the initial phases of training, and turned off afterwards.
  arXiv  Detail & Related papers  (2023-03-02T18:59:15Z)
• Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models [46.24479693469042]
  This paper shows that 1) pre-training loss cannot fully explain downstream performance and 2) flatness of the model is well-correlated with downstream performance where pre-training loss is not.
  arXiv  Detail & Related papers  (2022-10-25T17:45:36Z)
• An Empirical Investigation of the Role of Pre-training in Lifelong Learning [21.995593026269578]
  We show that generic pre-training implicitly alleviates the effects of catastrophic forgetting when learning multiple tasks sequentially. We study this phenomenon by analyzing the loss landscape, finding that pre-trained weights appear to ease forgetting by leading to wider minima.
  arXiv  Detail & Related papers  (2021-12-16T19:00:55Z)
• Mitigating Catastrophic Forgetting in Scheduled Sampling with Elastic Weight Consolidation in Neural Machine Translation [15.581515781839656]
  Autoregressive models trained with maximum likelihood estimation suffer from exposure bias. We propose using Elastic Weight Consolidation as trade-off between mitigating exposure bias and retaining output quality. Experiments on two IWSLT'14 translation tasks demonstrate that our approach alleviates catastrophic forgetting and significantly improves BLEU.
  arXiv  Detail & Related papers  (2021-09-13T20:37:58Z)
• Mixing between the Cross Entropy and the Expectation Loss Terms [89.30385901335323]
  Cross entropy loss tends to focus on hard to classify samples during training. We show that adding to the optimization goal the expectation loss helps the network to achieve better accuracy. Our experiments show that the new training protocol improves performance across a diverse set of classification domains.
  arXiv  Detail & Related papers  (2021-09-12T23:14:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.