Correction of Decoupled Weight Decay

• URL: http://arxiv.org/abs/2512.08217v1
• Date: Tue, 09 Dec 2025 03:52:54 GMT
• Title: Correction of Decoupled Weight Decay
• Abstract summary: We argue that decoupled weight decay $propto 2$ leads to stable weight norm based on simple assumption that updates become independent of steady state.<n>We show that decoupled weight decay $propto 2$ leads to stable weight and gradient norms and allows us to better control the training dynamics and improve the model performance.
• Score: 2.538209532048867
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decoupled weight decay, solely responsible for the performance advantage of AdamW over Adam, has long been set to proportional to learning rate $γ$ without questioning. Some researchers have recently challenged such assumption and argued that decoupled weight decay should be set $\propto γ^2$ instead based on orthogonality arguments at steady state. To the contrary, we find that eliminating the contribution of the perpendicular component of the update to the weight norm leads to little change to the training dynamics. Instead, we derive that decoupled weight decay $\propto γ^2$ results in stable weight norm based on the simple assumption that updates become independent of the weights at steady state, regardless of the nature of the optimizer. Based on the same assumption, we derive and empirically verify that the Total Update Contribution (TUC) of a minibatch under the Scion optimizer is better characterized by the momentum-dependent effective learning rate whose optimal value transfers and we show that decoupled weight decay $\propto γ^2$ leads to stable weight and gradient norms and allows us to better control the training dynamics and improve the model performance.

Related papers

• Scale Weight Decay and Train Better [0.0]
  We propose to scale weight decay by a fraction of the peak learning rate $/_max$.<n>We prove that this preserves the stationarity guarantees of the corresponding unregularized methods for both a gradient descent and a non-Euclidean Muon.<n>Using a steady-state analysis, we explain why under standard weight decay the weight norm shrinks steadily as training proceeds, whereas under scaled weight decay it settles to roughly constant value.
  arXiv  Detail & Related papers  (2026-07-26T17:55:40Z)
• Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors [110.10775872372047]
  We propose emphMagnitude-Direction-Direction (MD) Decoupling.<n>It factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates.<n>Across both Adam and Muon, MD Decoupling improves on well-tuned baselines, transfers the optimal LR across model width without retuning, and continues to help at scale on large Mixture-of-Experts models.
  arXiv  Detail & Related papers  (2026-06-24T15:40:26Z)
• Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization [67.29294986811314]
  Matrix baseds such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink when using standard constant weight decay.<n>We propose Hyperball, a simple wrapper that addresses this issue.<n>On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20-30% token equivalent speedup over weight decay baselines.
  arXiv  Detail & Related papers  (2026-06-15T16:09:02Z)
• Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics [0.0]
  We study why the Weibull weight-scale parameter $(t)$ grows, overshoots, and then relaxes during AdamW training.<n>Near saturation, alignment and decay approach balance, explaining the transition from weight-scale growth to relaxation.
  arXiv  Detail & Related papers  (2026-06-11T03:50:14Z)
• Bug or Feature$^2$: Weight Drift, Activation Sparsity and Spikes [53.726365933748134]
  We analyze a negative weight drift induced by the interaction between standard losses and positively biased activation functions.<n>We characterize the sparsity-accuracy tradeoff across 79 configurations and identify a sharp accuracy cliff above $sim$70% activation sparsity.
  arXiv  Detail & Related papers  (2026-05-17T21:29:20Z)
• Reinforcement Learning from Multi-Source Imperfect Preferences: Best-of-Both-Regimes Regret [71.69884486156359]
  We study episodic RL from emphmulti-source imperfect preferences through a cumulative imperfection budget.<n>We propose a unified algorithm with regret $tildeO(sqrtK/M+)$, which exhibits a best-of-both-regimes behavior.
  arXiv  Detail & Related papers  (2026-03-20T19:34:53Z)
• AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training [0.2578242050187029]
  AdamHuberDecay is a drop-in replacement for AdamW that substitutes the $ell$ penalty with a decoupled smooth Huber regularizer.<n>Experiments on GPT-2 and GPT-3 pre-training demonstrate that AdamHuberDecay converges 10-15% faster in wall-clock time.
  arXiv  Detail & Related papers  (2025-11-18T18:08:20Z)
• Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models [27.847140934456288]
  This paper proposes a new weight decay technique, Selective Projection Decay (SPD) SPD selectively imposes a strong penalty on certain layers while allowing others to change freely. When equipped with SPD, Adam consistently provides better in-distribution robustness and out-of-distribution performance on benchmarks.
  arXiv  Detail & Related papers  (2024-11-03T23:36:53Z)
• Large Continual Instruction Assistant [59.585544987096974]
  Continual Instruction Tuning (CIT) is adopted to instruct Large Models to follow human intent data by data.<n>Existing update gradient would heavily destroy the performance on previous datasets during CIT process.<n>We propose a general continual instruction tuning framework to address the challenge.
  arXiv  Detail & Related papers  (2024-10-08T11:24:59Z)
• Generalized Balancing Weights via Deep Neural Networks [0.0]
  Estimating causal effects from observational data is a central problem in many domains. We present generalized balancing weights, Neural Balancing Weights (NBW), to estimate the causal effects of an arbitrary mixture of discrete and continuous interventions.
  arXiv  Detail & Related papers  (2022-11-14T17:03:56Z)
• Improving Robustness with Adaptive Weight Decay [8.096469295357737]
  We propose adaptive weight decay, which automatically tunes the hyper- parameter iteration for weight decay during each training. We show that this simple modification can result in large improvements in robustness. This method has other desirable properties, such as less sensitivity to learning rate, and smaller weight norms.
  arXiv  Detail & Related papers  (2022-09-30T21:13:00Z)
• Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification [74.62203971625173]
  Imbalanced data pose challenges for deep learning based classification models. One of the most widely-used approaches for tackling imbalanced data is re-weighting. We propose a novel re-weighting method based on optimal transport (OT) from a distributional point of view.
  arXiv  Detail & Related papers  (2022-08-05T01:23:54Z)
• Long-Tailed Recognition via Weight Balancing [66.03068252811993]
  Naive training produces models that are biased toward common classes in terms of higher accuracy. We investigate three techniques to balance weights, L2-normalization, weight decay, and MaxNorm. Our approach achieves the state-of-the-art accuracy on five standard benchmarks.
  arXiv  Detail & Related papers  (2022-03-27T03:26:31Z)
• On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective [90.39123717733334]
  We present the first practical scheduler for weight decay, called the Scheduled Weight Decay (SWD) method. Our experiments also support that SWD indeed mitigates large gradient norms and often significantly outperforms the conventional constant weight decay strategy for Adaptive Moment Estimation (Adam)
  arXiv  Detail & Related papers  (2020-11-23T00:39:49Z)
• Optimal Variance Control of the Score Function Gradient Estimator for Importance Weighted Bounds [12.75471887147565]
  This paper introduces novel results for the score function gradient estimator of the importance weighted variational bound (IWAE) We prove that in the limit of large $K$ one can choose the control variate such that the Signal-to-Noise ratio (SNR) of the estimator grows as $sqrtK$.
  arXiv  Detail & Related papers  (2020-08-05T08:41:46Z)
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights [53.8489656709356]
  Normalization techniques are a boon for modern deep learning. It is often overlooked, however, that the additional introduction of momentum results in a rapid reduction in effective step sizes for scale-invariant weights. In this paper, we verify that the widely-adopted combination of the two ingredients lead to the premature decay of effective step sizes and sub-optimal model performances.
  arXiv  Detail & Related papers  (2020-06-15T08:35:15Z)
• Understanding the Difficulty of Training Transformers [120.99980924577787]
  We show that unbalanced gradients are not the root cause of the instability of training. We propose Admin to stabilize the early stage's training and unleash its full potential in the late stage.
  arXiv  Detail & Related papers  (2020-04-17T13:59:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.