On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective

• URL: http://arxiv.org/abs/2011.11152v6
• Date: Fri, 16 Aug 2024 10:36:24 GMT
• Title: On the Overlooked Pitfalls of Weight Decay and How to Mitigate Them: A Gradient-Norm Perspective
• Authors: Zeke Xie, Zhiqiang Xu, Jingzhao Zhang, Issei Sato, Masashi Sugiyama,
• Abstract summary: We present the first practical scheduler for weight decay, called the Scheduled Weight Decay (SWD) method. Our experiments also support that SWD indeed mitigates large gradient norms and often significantly outperforms the conventional constant weight decay strategy for Adaptive Moment Estimation (Adam)
• Score: 90.39123717733334
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Weight decay is a simple yet powerful regularization technique that has been very widely used in training of deep neural networks (DNNs). While weight decay has attracted much attention, previous studies fail to discover some overlooked pitfalls on large gradient norms resulted by weight decay. In this paper, we discover that, weight decay can unfortunately lead to large gradient norms at the final phase (or the terminated solution) of training, which often indicates bad convergence and poor generalization. To mitigate the gradient-norm-centered pitfalls, we present the first practical scheduler for weight decay, called the Scheduled Weight Decay (SWD) method that can dynamically adjust the weight decay strength according to the gradient norm and significantly penalize large gradient norms during training. Our experiments also support that SWD indeed mitigates large gradient norms and often significantly outperforms the conventional constant weight decay strategy for Adaptive Moment Estimation (Adam).

Related papers

• Rethinking Weight Decay for Robust Fine-Tuning of Foundation Models [27.847140934456288]
  This paper proposes a new weight decay technique, Selective Projection Decay (SPD) SPD selectively imposes a strong penalty on certain layers while allowing others to change freely. When equipped with SPD, Adam consistently provides better in-distribution robustness and out-of-distribution performance on benchmarks.
  arXiv  Detail & Related papers  (2024-11-03T23:36:53Z)
• Why Do We Need Weight Decay in Modern Deep Learning? [24.81634291051533]
  Weight decay is a technique for training state-of-the-art deep networks from image classification to large language models. In this work, we highlight that the role of weight decay in modern deep learning is different from its regularization effect studied in classical learning theory. For deep networks on vision tasks trained with multipass SGD, we show how weight decay modifies the optimization dynamics enhancing the implicit regularization of SGD.
  arXiv  Detail & Related papers  (2023-10-06T17:58:21Z)
• FedNAR: Federated Optimization with Normalized Annealing Regularization [54.42032094044368]
  We explore the choices of weight decay and identify that weight decay value appreciably influences the convergence of existing FL algorithms. We develop Federated optimization with Normalized Annealing Regularization (FedNAR), a plug-in that can be seamlessly integrated into any existing FL algorithms.
  arXiv  Detail & Related papers  (2023-10-04T21:11:40Z)
• PathProx: A Proximal Gradient Algorithm for Weight Decay Regularized Deep Neural Networks [25.114642281756495]
  Weight decay is one of the most widely used forms of regularization in deep learning. This paper argues that gradient descent may be an inefficient algorithm for this objective. For neural networks with ReLU activations, solutions to the weight decay objective are equivalent to those of a different objective.
  arXiv  Detail & Related papers  (2022-10-06T17:22:40Z)
• SGD and Weight Decay Secretly Minimize the Rank of Your Neural Network [8.79431718760617]
  Training with mini-batch SGD and weight decay induces a bias toward rank minimization in weight matrices. We show that this bias becomes more pronounced with smaller batch sizes, higher learning rates, or stronger weight decay. We empirically explore the connection between this bias and generalization, finding that it has a marginal effect on the test performance.
  arXiv  Detail & Related papers  (2022-06-12T17:06:35Z)
• Long-Tailed Recognition via Weight Balancing [66.03068252811993]
  Naive training produces models that are biased toward common classes in terms of higher accuracy. We investigate three techniques to balance weights, L2-normalization, weight decay, and MaxNorm. Our approach achieves the state-of-the-art accuracy on five standard benchmarks.
  arXiv  Detail & Related papers  (2022-03-27T03:26:31Z)
• The Implicit Biases of Stochastic Gradient Descent on Deep Neural Networks with Batch Normalization [44.30960913470372]
  Deep neural networks with batch normalization (BN-DNNs) are invariant to weight rescaling due to their normalization operations. We investigate the implicit biases of gradient descent (SGD) on BN-DNNs to provide a theoretical explanation for the efficacy of weight decay.
  arXiv  Detail & Related papers  (2021-02-06T03:40:20Z)
• Improve Generalization and Robustness of Neural Networks via Weight Scale Shifting Invariant Regularizations [52.493315075385325]
  We show that a family of regularizers, including weight decay, is ineffective at penalizing the intrinsic norms of weights for networks with homogeneous activation functions. We propose an improved regularizer that is invariant to weight scale shifting and thus effectively constrains the intrinsic norm of a neural network.
  arXiv  Detail & Related papers  (2020-08-07T02:55:28Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)
• On the Generalization of Stochastic Gradient Descent with Momentum [84.54924994010703]
  momentum-based accelerated variants of gradient descent (SGD) are widely used when training machine learning models. We first show that there exists a convex loss function for which the stability gap for multiple epochs of SGD with standard heavy-ball momentum (SGDM) becomes unbounded. For smooth Lipschitz loss functions, we analyze a modified momentum-based update rule, i.e., SGD with early momentum (SGDEM) under a broad range of step-sizes.
  arXiv  Detail & Related papers  (2018-09-12T17:02:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.