SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training

• URL: http://arxiv.org/abs/2605.27541v1
• Date: Tue, 26 May 2026 18:14:19 GMT
• Title: SparseOpt: Addressing Normalization-induced Gradient Skew in Sparse Training
• Abstract summary: Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology.<n>We show that Batch Normalization (BN) adversely affects sparse training, and propose SparseOpt, a sparsity-aware method to address this.
• Score: 30.444606429625377
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Dynamic Sparse Training (DST) methods train neural networks by maintaining sparsity while dynamically adapting the network topology. Despite the promise of reduced computation, DST methods converge significantly slower than dense training, often requiring comparable training time to achieve similar accuracy. We demonstrate both analytically and empirically that Batch Normalization (BN) adversely affects sparse training, and propose SparseOpt, a sparsity-aware optimizer, to address this. Experiments on ResNet models across CIFAR-100 and ImageNet demonstrate consistently faster convergence and improved generalization with our proposed method. Our work highlights the limitations of current normalization layers in sparse training and provides the first systematic study of the interaction between Batch Normalization, sparse layers, and DST, taking a significant step toward making DST practically competitive with dense training.

Related papers

• Memory-Efficient LLM Training with Dynamic Sparsity: From Stability to Practical Scaling [38.73262288439019]
  Dynamic Sparse Training (DST) offers a promising paradigm for improving the training and inference efficiency of deep neural networks.<n>In large language model training, DST can suffer from optimization instability, manifested as loss spikes after topology updates.<n>We propose Sparse Memory-Efficient Training (SMET), which stabilizes DST with warm-up and improves training progress through density-aware learning-rate scaling.
  arXiv  Detail & Related papers  (2026-05-30T20:47:45Z)
• Sparse Spectral Training and Inference on Euclidean and Hyperbolic Neural Networks [9.96381061452642]
  Low-Rank Adaptation (LoRA) and ReLoRA face challenges with their low-rank structure.<n>We propose Sparse Spectral Training (SST) to optimize memory usage for pre-training.<n>SST reduces the perplexity gap between other low-rank methods and full-rank training by 97.4%.
  arXiv  Detail & Related papers  (2024-05-24T11:59:41Z)
• Accurate Neural Network Pruning Requires Rethinking Sparse Optimization [87.90654868505518]
  We show the impact of high sparsity on model training using the standard computer vision and natural language processing sparsity benchmarks. We provide new approaches for mitigating this issue for both sparse pre-training of vision models and sparse fine-tuning of language models.
  arXiv  Detail & Related papers  (2023-08-03T21:49:14Z)
• Dynamic Sparse Training with Structured Sparsity [11.778353786208765]
  Dynamic Sparse Training (DST) methods achieve state-of-the-art results in sparse neural network training. We propose a sparse-to-sparse DST method, Structured RigL (SRigL), to learn a variant of fine-grained structured N:M sparsity. We demonstrate a real-world acceleration of 3.4x/2.5x on CPU for online inference and 1.7x/13.0x on GPU for inference with a batch size of 256.
  arXiv  Detail & Related papers  (2023-05-03T17:48:55Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Online Training Through Time for Spiking Neural Networks [66.7744060103562]
  Spiking neural networks (SNNs) are promising brain-inspired energy-efficient models. Recent progress in training methods has enabled successful deep SNNs on large-scale tasks with low latency. We propose online training through time (OTTT) for SNNs, which is derived from BPTT to enable forward-in-time learning.
  arXiv  Detail & Related papers  (2022-10-09T07:47:56Z)
• Distributed Adversarial Training to Robustify Deep Neural Networks at Scale [100.19539096465101]
  Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification. To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training. We propose a large-batch adversarial training framework implemented over multiple machines.
  arXiv  Detail & Related papers  (2022-06-13T15:39:43Z)
• Do We Actually Need Dense Over-Parameterization? In-Time Over-Parameterization in Sparse Training [16.81321230135317]
  We propose the concept of In-Time Over- Resibilityization (ITOP) in sparse training. ITOP closes the gap in the express between sparse training and dense training. We present a series of experiments to support our conjecture and achieve the state-of-the-art sparse training performance.
  arXiv  Detail & Related papers  (2021-02-04T20:59:31Z)
• Selfish Sparse RNN Training [13.165729746380816]
  We propose an approach to train sparse RNNs with a fixed parameter count in one single run, without compromising performance. We achieve state-of-the-art sparse training results with various datasets on Penn TreeBank and Wikitext-2.
  arXiv  Detail & Related papers  (2021-01-22T10:45:40Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• Large Batch Training Does Not Need Warmup [111.07680619360528]
  Training deep neural networks using a large batch size has shown promising results and benefits many real-world applications. In this paper, we propose a novel Complete Layer-wise Adaptive Rate Scaling (CLARS) algorithm for large-batch training. Based on our analysis, we bridge the gap and illustrate the theoretical insights for three popular large-batch training techniques.
  arXiv  Detail & Related papers  (2020-02-04T23:03:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.