Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling

• URL: http://arxiv.org/abs/2606.00888v1
• Date: Sat, 30 May 2026 20:47:45 GMT
• Title: Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling
• Abstract summary: Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks.<n>In large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates.<n>We propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with warm-up and improves training progress through density-aware learning-rate scaling.
• Score: 38.73262288439019
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks; however, we find that in large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates. In this work, we show that the naive use of standard Adam-based optimizers leads to a cold-start issue for newly regrown parameters, resulting in excessively large updates and disrupted training dynamics. To address this issue, we propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with optimizer warm-up and improves training progress through density-aware learning-rate scaling. SMET further reduces memory consumption by storing gradients and optimizer states only for active parameters. We provide a theoretical analysis of the update behaviors under SMET, showing improved optimization stability. Extensive experiments demonstrate that SMET enables stable, scalable, and memory-efficient sparse pre-training of LLMs, paving the way for sparse training as a practical alternative to dense training. Our code is publicly available at: https://github.com/QiaoXiao7282/SMET.

Related papers

• SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training [30.444606429625377]
  Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology.<n>We show that Batch Normalization (BN) adversely affects sparse training, and propose SparseOpt, a sparsity-aware method to address this.
  arXiv  Detail & Related papers  (2026-05-26T18:14:19Z)
• Beyond Freezing: Sparse Tuning Enhances Plasticity in Continual Learning with Pre-Trained Models [10.904981532789824]
  Continual Learning with Pre-trained Models holds great promise for efficient adaptation across sequential tasks.<n>Existing approaches freeze PTMs and rely on auxiliary modules like prompts or adapters.<n>We propose Mutual Information-guided Sparse Tuning (MIST), a plug-and-play method that selectively updates a small subset of PTM parameters.
  arXiv  Detail & Related papers  (2025-05-26T13:09:25Z)
• SubTrack++ : Gradient Subspace Tracking for Scalable LLM Training [4.804685266333252]
  Training large language models (LLMs) is highly resource-intensive due to their massive number of parameters and the overhead of states.<n>We propose SubTrack++ that leverages Grassmannian subspace tracking combined with projection-aware gradients.<n>Our method demonstrates SOTA convergence by exploiting Grassmannian geometry, reducing pre-training wall-time by up to 65% and fine-tuning time by 36% compared to existing SOTA methods.
  arXiv  Detail & Related papers  (2025-02-03T18:13:52Z)
• SPAM: Spike-Aware Adam with Momentum Reset for Stable LLM Training [60.9776082805359]
  Large Language Models (LLMs) have demonstrated exceptional performance across diverse tasks, yet their training remains highly resource-intensive and susceptible to training instability.<n>This paper presents a comprehensive investigation into gradient spikes observed during LLM training, revealing their prevalence across multiple architectures and datasets.<n>We propose Spike-Aware Adam with Momentum Reset, a novel designed to counteract gradient spikes through momentum reset and spike-aware clipping.
  arXiv  Detail & Related papers  (2025-01-12T15:21:22Z)
• Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks [9.96381061452642]
  Low-Rank Adaptation (LoRA) and ReLoRA face challenges with their low-rank structure.<n>We propose Sparse Spectral Training (SST) to optimize memory usage for pre-training.<n>SST reduces the perplexity gap between other low-rank methods and full-rank training by 97.4%.
  arXiv  Detail & Related papers  (2024-05-24T11:59:41Z)
• AdaLomo: Low-memory Optimization with Adaptive Learning Rate [59.64965955386855]
  We introduce low-memory optimization with adaptive learning rate (AdaLomo) for large language models. AdaLomo results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.
  arXiv  Detail & Related papers  (2023-10-16T09:04:28Z)
• Online Convolutional Re-parameterization [51.97831675242173]
  We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
  arXiv  Detail & Related papers  (2022-04-02T09:50:19Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• Dynamic Scale Training for Object Detection [111.33112051962514]
  We propose a Dynamic Scale Training paradigm (abbreviated as DST) to mitigate scale variation challenge in object detection. Experimental results demonstrate the efficacy of our proposed DST towards scale variation handling. It does not introduce inference overhead and could serve as a free lunch for general detection configurations.
  arXiv  Detail & Related papers  (2020-04-26T16:48:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.