Does Weight Decay Enhance Training Stability?

• URL: http://arxiv.org/abs/2605.16622v1
• Date: Fri, 15 May 2026 20:43:26 GMT
• Title: Does Weight Decay Enhance Training Stability?
• Abstract summary: We consider how weight decay affects the parameter-space dynamics and loss sharpness.<n>We identify the global alignment of the parameter vector and the sharpness gradient as the mechanistic driver of the phase transition.<n>This shows that curvature thresholds obtained from convex/quadratics may not be reliable stability diagnostics under regularization.
• Score: 0.5799785223420272
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In modern deep learning, weight decay is often credited with "stabilizing" training dynamics, diverging from its classical role as a static regularization penalty. We investigate a fundamental question: *does weight decay stabilize training dynamics, and if so, through which mechanism?* Indeed, training stability is understood through different but related notions in the literature. We consider how weight decay affects the parameter-space dynamics and loss sharpness by analyzing its effects at the \emph{Edge of Stability} (EoS). We show that weight decay robustly slows *progressive sharpening}. Furthermore, we uncover a striking architecture-dependent phase transition. In CNNs, weight decay dampens the oscillations at the EoS, while in MLPs, increasing weight decay causes a phase transition in which the sharpness stabilizes at a threshold significantly below the theoretical $\frac{2}η$ boundary. We develop a mathematical framework that accurately models these phenomena and identify the global alignment of the parameter vector and the sharpness gradient as the mechanistic driver of the phase transition. Importantly, we show that these phenomena translate into stability in terms of search in function-space (NTK). Last, this shows that curvature thresholds obtained from convex/quadratic heuristics may not be reliable stability diagnostics under regularization.

Related papers

• Weight-norm Criticality: A Mechanism for Loss Spikes Induced by the Normalization and Weight Decay [12.529130624059597]
  We argue that, in practical deep neural network training, there is an additional and often overlooked emphweight-norm criticality<n>This criticality is induced by the interaction between normalization (which introduces scale-invariant components) and weight decay.<n>Our work provides a new mechanistic understanding of loss spikes through the lens of emphweight-norm criticality
  arXiv  Detail & Related papers  (2026-07-23T07:39:03Z)
• SGD at the Edge of Stability: Stochastic Stabilization with Large Learning Rates [2.104588729923885]
  Modern deep learning has been shown to operate at the edge of stability.<n>We provide convergence guarantees for Gradient Descent (SGD) applied to the multiclass cross-entropy loss.<n> Experiments validate our theoretical findings and illustrate the benefits of SGD in the large-stepsize regime.
  arXiv  Detail & Related papers  (2026-06-29T21:32:58Z)
• Taming Curvature: Architecture Warm-Up for Stable Transformer Training [52.6838951211596]
  We introduce a fast online estimator of the largest (preconditioned) Hessian eigenvalue (i.e., curvature) based on a warm-started variant for power with Hessian-vector products.<n>We find that training instabilities coincide with surges in preconditioned curvature and that curvature grows with depth.<n> Experiments on large Transformers validate that our approach enables efficient curvature tracking and reduces instabilities.
  arXiv  Detail & Related papers  (2026-06-15T14:16:56Z)
• Gradient descent at the Edge of Stability: free energy model and kinetic description of the two-layer network [0.0]
  We study the dynamics of gradient descent in the Edge of Stability regime.<n>We propose a continuous-time effective model that tracks the evolution of the average trajectory.<n>We show that this equation can be interpreted as a Wasserstein-2 gradient flow of a macroscopic free energy.
  arXiv  Detail & Related papers  (2026-06-03T18:11:33Z)
• Detecting Deepfakes via Hamiltonian Dynamics [58.881686693914446]
  We propose a new perspective on deepfake detection: moving from static pattern recognition to dynamical stability analysis.<n>Our approach is motivated by physics-inspired priors.<n>Hamiltonian Action Anomaly Detection (HAAD) outperforms evaluated state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-05-06T01:55:38Z)
• Learning to Forget: Continual Learning with Adaptive Weight Decay [30.558842015806363]
  Continual learning agents with finite capacity must balance acquiring new knowledge with retaining the old.<n>Weight decay, viewed as a mechanism for forgetting, can serve this role by gradually discarding information stored in the weights.<n>We introduce Forgetting through Adaptive Decay (FADE), which adapts per- parameter weight decay rates online via approximate meta-gradient descent.
  arXiv  Detail & Related papers  (2026-04-29T18:01:13Z)
• Stability as a Liability:Systematic Breakdown of Linguistic Structure in LLMs [5.96875296117642]
  We show that stable parameter trajectories lead stationary solutions to minimize the forward KL divergence to the empirical distribution.<n>We empirically validate this effect using a controlled feedback-based training framework.<n>It indicates that optimization stability and generative expressivity are not inherently aligned, and that stability alone is an insufficient indicator of generative quality.
  arXiv  Detail & Related papers  (2026-01-26T15:34:50Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability [40.17821914923602]
  We show that gradient descent at the edge of stability implicitly follows projected gradient descent (PGD) under the constraint $S(theta) le 2/eta$. Our analysis provides precise predictions for the loss, sharpness, and deviation from the PGD trajectory throughout training.
  arXiv  Detail & Related papers  (2022-09-30T17:15:12Z)
• A PDE-based Explanation of Extreme Numerical Sensitivities and Edge of Stability in Training Neural Networks [12.355137704908042]
  We show restrained numerical instabilities in current training practices of deep networks with gradient descent (SGD) We do this by presenting a theoretical framework using numerical analysis of partial differential equations (PDE), and analyzing the gradient descent PDE of convolutional neural networks (CNNs) We show this is a consequence of the non-linear PDE associated with the descent of the CNN, whose local linearization changes when over-driving the step size of the discretization resulting in a stabilizing effect.
  arXiv  Detail & Related papers  (2022-06-04T14:54:05Z)
• On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective [90.39123717733334]
  We present the first practical scheduler for weight decay, called the Scheduled Weight Decay (SWD) method. Our experiments also support that SWD indeed mitigates large gradient norms and often significantly outperforms the conventional constant weight decay strategy for Adaptive Moment Estimation (Adam)
  arXiv  Detail & Related papers  (2020-11-23T00:39:49Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)
• Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent [55.85456985750134]
  We introduce a new stability measure called on-average model stability, for which we develop novel bounds controlled by the risks of SGD iterates. This yields generalization bounds depending on the behavior of the best model, and leads to the first-ever-known fast bounds in the low-noise setting. To our best knowledge, this gives the firstever-known stability and generalization for SGD with even non-differentiable loss functions.
  arXiv  Detail & Related papers  (2020-06-15T06:30:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.