Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting

• URL: http://arxiv.org/abs/2605.02105v1
• Date: Mon, 04 May 2026 00:02:23 GMT
• Title: Sharpness-Aware Pretraining Mitigates Catastrophic Forgetting
• Abstract summary: We study three pretraining optimization approaches that bias optimization toward flatter minima.<n>Across model sizes ranging from 20M to 150M parameters, we find that these interventions consistently improve downstream performance after post-training.<n>These principles hold at scale: a short SAM mid-training phase applied to an existing OLMo-2-1B checkpoint reduces forgetting by 31% after MetaMath post-training and by 40% after 4-bit quantization.
• Score: 34.224490336271806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pretraining optimizers are tuned to produce the strongest possible base model, on the assumption that a stronger starting point yields a stronger model after subsequent changes like post-training and quantization. This overlooks the geometry of the base model which controls how much of the base model's capabilities survive subsequent parameter updates. We study three pretraining optimization approaches that bias optimization toward flatter minima: Sharpness-Aware Minimization (SAM), large learning rates, and shortened learning rate annealing periods. Across model sizes ranging from 20M to 150M parameters, we find that these interventions consistently improve downstream performance after post-training on five common datasets with up to 80% less forgetting. These principles hold at scale: a short SAM mid-training phase applied to an existing OLMo-2-1B checkpoint reduces forgetting by 31% after MetaMath post-training and by 40% after 4-bit quantization.

Related papers

• Chain-of-Models Pre-Training: Rethinking Training Acceleration of Vision Foundation Models [16.42169130504086]
  Chain-of-Models Pre-Training (CoM-PT) is a performance-lossless training acceleration method for vision foundation models.<n>CoM-PT is designed to accelerate the training pipeline at the model family level, scaling efficiently as the model family expands.<n>It is extensively validated across 45 datasets spanning zero-shot and fine-tuning tasks.
  arXiv  Detail & Related papers  (2026-04-14T07:26:23Z)
• ReMiT: RL-Guided Mid-Training for Iterative LLM Evolution [49.496216822640974]
  We analyze training dynamics and identify the mid-training phase as a critical turning point for model capabilities.<n>We introduce ReMiT (Reinforcement Learning-Guided Mid-Training), which prioritizes tokens during the mid-training phase, prioritizing those pivotal for reasoning.
  arXiv  Detail & Related papers  (2026-02-03T04:04:41Z)
• IMU-1: Sample-Efficient Pre-training of Small Language Models [0.0]
  We present IMU-1, a 430M- parameter language model trained on 72B tokens that approaches the benchmark performance of models trained on 56x more data.<n>We describe a validated training recipe combining recent architectural interventions (QK-norm attention, per-head gating, value residuals, LayerNorm scaling) with optimization advances.
  arXiv  Detail & Related papers  (2026-01-25T21:24:15Z)
• CAST: Continuous and Differentiable Semi-Structured Sparsity-Aware Training for Large Language Models [27.682531424487564]
  Sparsity-aware training is an effective approach for transforming large language models into hardware-friendly sparse patterns.<n>We propose Continuous Adaptive Sparse Trainer (CAST), a continuous and differentiable sparsity-aware training framework for sparse models.<n>Our results demonstrate significant improvements over previous state-of-the-art methods in both perplexity and zero-shot accuracy with minimal training resources.
  arXiv  Detail & Related papers  (2025-09-30T09:28:47Z)
• Beyond Outliers: A Study of Optimizers Under Quantization [82.75879062804955]
  We study impact of choice on model robustness under quantization.<n>We evaluate how model performance degrades when trained with different baselines.<n>We derive scaling laws for quantization-aware training under different parameters.
  arXiv  Detail & Related papers  (2025-09-27T21:15:22Z)
• Overtrained Language Models Are Harder to Fine-Tune [64.44743256512237]
  Large language models are pre-trained on ever-growing token budgets.<n>We show that extended pre-training can make models harder to fine-tune, leading to degraded final performance.
  arXiv  Detail & Related papers  (2025-03-24T23:11:56Z)
• The Journey Matters: Average Parameter Count over Pre-training Unifies Sparse and Dense Scaling Laws [51.608402959163925]
  We present the first systematic exploration of optimal sparse pre-training configurations for large language models.<n>We find that initiating pruning at 25% of total training compute and concluding at 75% achieves near-optimal final evaluation loss.<n>We propose a new scaling law that modifies the Chinchilla scaling law to use the average parameter count over pre-training.
  arXiv  Detail & Related papers  (2025-01-21T20:23:22Z)
• Efficient Continual Pre-training by Mitigating the Stability Gap [68.49269649759005]
  We study the behavior of Large Language Models (LLMs) during continual pre-training. We propose three effective strategies to enhance LLM performance within a fixed compute budget. Our strategies improve the average medical task performance of the OpenLlama-3B model from 36.2% to 40.7% with only 40% of the original training budget.
  arXiv  Detail & Related papers  (2024-06-21T02:28:37Z)
• The Fine Line: Navigating Large Language Model Pretraining with Down-streaming Capability Analysis [27.310894780313618]
  This paper undertakes a comprehensive comparison of model capabilities at various pretraining intermediate checkpoints. We confirm that specific downstream metrics exhibit similar training dynamics across models of different sizes. In addition to our core findings, we've reproduced Amber and OpenLLaMA, releasing their intermediate checkpoints.
  arXiv  Detail & Related papers  (2024-04-01T16:00:01Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.