Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay

• URL: http://arxiv.org/abs/2607.21005v1
• Date: Thu, 23 Jul 2026 07:39:03 GMT
• Title: Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay
• Abstract summary: We argue that, in practical deep neural network training, there is an additional and often overlooked emphweight-norm criticality<n>This criticality is induced by the interaction between normalization (which introduces scale-invariant components) and weight decay.<n>Our work provides a new mechanistic understanding of loss spikes through the lens of emphweight-norm criticality
• Score: 12.529130624059597
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most explanations of training instability focus on \emph{learning-rate criticality}, typically characterized by the Edge of Stability, beyond which optimization becomes unstable. We argue that, in practical deep neural network training, there is an additional and often overlooked \emph{weight-norm criticality}. This criticality is induced by the interaction between normalization (which introduces scale-invariant components) and weight decay (which persistently shrinks parameter norms). As the weight decay coefficient increases, the norms of scale-invariant weights are progressively driven toward zero. Meanwhile, the sharpness of the loss landscape increases rapidly, destabilizing the optimization dynamics and resulting in abrupt loss spikes. This perspective provides a rationale for why weight penalties can improve generalization yet cannot be made arbitrarily strong: excessive decay drives scale-invariant weight norms past a critical boundary and destabilizes training. Our work provides a new mechanistic understanding of loss spikes through the lens of \emph{weight-norm criticality}. Moreover, \emph{weight-norm criticality} yields testable predictions that we validate empirically in networks with scale-invariant components, providing empirical support for the proposed mechanism.

Related papers

• When Does Scale-Invariant Optimization Become Unstable? An Exact Schedule Law with Weight Decay [6.625270757109195]
  We show that a single quantity captures all schedule and decay forcing, while norm growth induces an opposing geometric self-quenching effect.<n>This yields a sharp boundary that cleanly separates contraction- and expansion-dominated effective learning rate regimes.
  arXiv  Detail & Related papers  (2026-09-08T17:46:17Z)
• Does Weight Decay Enhance Training Stability? [0.5799785223420272]
  We consider how weight decay affects the parameter-space dynamics and loss sharpness.<n>We identify the global alignment of the parameter vector and the sharpness gradient as the mechanistic driver of the phase transition.<n>This shows that curvature thresholds obtained from convex/quadratics may not be reliable stability diagnostics under regularization.
  arXiv  Detail & Related papers  (2026-05-15T20:43:26Z)
• Relative Kinetic Utility for Reasoning-Aware Structural Pruning in Large Language Models [0.0]
  We propose a novel framework that elevates discrete pruning to a continuous kinetic integral over the depth manifold of a model.<n>Experiments on Qwen-2.5-7B and LLaMA-3-8B improve performance in the high-sparsity regime around 40%.<n>RKU attains 13.34% accuracy on GSM8K at 40% sparsity, outperforming the strongest baseline.
  arXiv  Detail & Related papers  (2026-05-09T15:47:53Z)
• Random-Matrix-Induced Simplicity Bias in Over-parameterized Variational Quantum Circuits [72.0643009153473]
  We show that expressive variational ansatze enter a Haar-like universality class in which both observable expectation values and parameter gradients concentrate exponentially with system size.<n>As a consequence, the hypothesis class induced by such circuits collapses with high probability to a narrow family of near-constant functions.<n>We further show that this collapse is not unavoidable: tensor-structured VQCs, including tensor-network-based and tensor-hypernetwork parameterizations, lie outside the Haar-like universality class.
  arXiv  Detail & Related papers  (2026-01-05T08:04:33Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Non-Singular Adversarial Robustness of Neural Networks [58.731070632586594]
  Adrial robustness has become an emerging challenge for neural network owing to its over-sensitivity to small input perturbations. We formalize the notion of non-singular adversarial robustness for neural networks through the lens of joint perturbations to data inputs as well as model weights.
  arXiv  Detail & Related papers  (2021-02-23T20:59:30Z)
• Explicit regularization and implicit bias in deep network classifiers trained with the square loss [2.8935588665357077]
  Deep ReLU networks trained with the square loss have been observed to perform well in classification tasks. We show that convergence to a solution with the absolute minimum norm is expected when normalization techniques are used together with Weight Decay.
  arXiv  Detail & Related papers  (2020-12-31T21:07:56Z)
• On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective [90.39123717733334]
  We present the first practical scheduler for weight decay, called the Scheduled Weight Decay (SWD) method. Our experiments also support that SWD indeed mitigates large gradient norms and often significantly outperforms the conventional constant weight decay strategy for Adaptive Moment Estimation (Adam)
  arXiv  Detail & Related papers  (2020-11-23T00:39:49Z)
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations [52.493315075385325]
  We show that a family of regularizers, including weight decay, is ineffective at penalizing the intrinsic norms of weights for networks with homogeneous activation functions. We propose an improved regularizer that is invariant to weight scale shifting and thus effectively constrains the intrinsic norm of a neural network.
  arXiv  Detail & Related papers  (2020-08-07T02:55:28Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.