A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms

• URL: http://arxiv.org/abs/2009.06125v2
• Date: Wed, 29 Sep 2021 23:55:28 GMT
• Title: A Qualitative Study of the Dynamic Behavior for Adaptive Gradient Algorithms
• Abstract summary: RMSprop and Adam algorithms are studied through a combination of careful numerical experiments and theoretical explanations. Three types of qualitative features are observed in the training loss curve: fast initial convergence, oscillations, and large spikes in the late phase.
• Score: 12.865834066050427
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The dynamic behavior of RMSprop and Adam algorithms is studied through a combination of careful numerical experiments and theoretical explanations. Three types of qualitative features are observed in the training loss curve: fast initial convergence, oscillations, and large spikes in the late phase. The sign gradient descent (signGD) flow, which is the limit of Adam when taking the learning rate to 0 while keeping the momentum parameters fixed, is used to explain the fast initial convergence. For the late phase of Adam, three different types of qualitative patterns are observed depending on the choice of the hyper-parameters: oscillations, spikes, and divergence. In particular, Adam converges much smoother and even faster when the values of the two momentum factors are close to each other. This observation is particularly important for scientific computing tasks, for which the training process usually proceeds into the high precision regime.

Related papers

• Balancing Learning Rates Across Layers: Exact Two-Step Dynamics and Optimal Scaling in Linear Neural Networks [40.02433292363059]
  We study optimal learning-rate selection in two-layer and three-layer linear neural networks trained to learn linear target functions.<n>We derive the exact closed-form expressions for the gradients and test loss after one and two steps of gradient descent.
  arXiv  Detail & Related papers  (2026-05-29T20:28:52Z)
• Convergence of Steepest Descent and Adam under Non-Uniform Smoothness [9.95160888054888]
  We show that for logistic regression on separable data, RMSProp and Adam converge linearly and is provably faster than GD.<n>We also show that for a class of two-layer neural networks on separable data, RMSProp and Adam can converge at a linear rate with a constant step-size and momentum parameter.
  arXiv  Detail & Related papers  (2026-05-28T23:05:45Z)
• Towards Understanding Adam Convergence on Highly Degenerate Polynomials [12.224244942795695]
  We study the "natural" auto-convergence properties of Adam.<n>We find a class of highly degenerates where Adam converges automatically without additional schedulers.<n>We prove that Adam degenerates local linear convergence on these functions, significantly outperforming the sub- convergence of Gradient Descent and Momentum.
  arXiv  Detail & Related papers  (2026-03-10T12:30:20Z)
• Similarity Matching Networks: Hebbian Learning and Convergence Over Multiple Time Scales [5.093257685701887]
  We consider and analyze the emphsimilarity matching network for principal subspace projection.<n>By leveraging a multilevel optimization framework, we prove convergence of the dynamics in the offline setting.
  arXiv  Detail & Related papers  (2025-06-06T14:46:22Z)
• Dynamical Decoupling of Generalization and Overfitting in Large Two-Layer Networks [12.061229162870513]
  We study the training dynamics of two-layer neural networks. We find several new phenomena in the training dynamics. These include the emergence of a slow time scale associated with the growth in Gaussian/Rademacher complexity.
  arXiv  Detail & Related papers  (2025-02-28T17:45:26Z)
• Dynamic Decoupling of Placid Terminal Attractor-based Gradient Descent Algorithm [56.06235614890066]
  Gradient descent (GD) and gradient descent (SGD) have been widely used in a number of application domains. This paper carefully analyzes the dynamics of GD based on the terminal attractor at different stages of its gradient flow.
  arXiv  Detail & Related papers  (2024-09-10T14:15:56Z)
• Conjugate-Gradient-like Based Adaptive Moment Estimation Optimization Algorithm for Deep Learning [2.695991050833627]
  We propose a new optimization algorithm named CG-like-Adam for deep learning. Specifically, both the first-order and the second-order moment estimation of generic Adam are replaced by the conjugate-gradient-like. Numerical experiments show the superiority of the proposed algorithm based on the CIFAR10/100 dataset.
  arXiv  Detail & Related papers  (2024-04-02T07:57:17Z)
• Adaptive Federated Learning Over the Air [108.62635460744109]
  We propose a federated version of adaptive gradient methods, particularly AdaGrad and Adam, within the framework of over-the-air model training. Our analysis shows that the AdaGrad-based training algorithm converges to a stationary point at the rate of $mathcalO( ln(T) / T 1 - frac1alpha ).
  arXiv  Detail & Related papers  (2024-03-11T09:10:37Z)
• Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality [54.20763128054692]
  We study the dynamics of gradient flow for training a multi-head softmax attention model for in-context learning of multi-task linear regression. We prove that an interesting "task allocation" phenomenon emerges during the gradient flow dynamics.
  arXiv  Detail & Related papers  (2024-02-29T18:43:52Z)
• A Bootstrap Algorithm for Fast Supervised Learning [0.0]
  Training a neural network (NN) typically relies on some type of curve-following method, such as gradient descent (and gradient descent (SGD)), ADADELTA, ADAM or limited memory algorithms. Convergence for these algorithms usually relies on having access to a large quantity of observations in order to achieve a high level of accuracy and, with certain classes of functions, these algorithms could take multiple epochs of data points to catch on. Herein, a different technique with the potential of achieving dramatically better speeds of convergence is explored: it does not curve-follow but rather relies on 'decoupling' hidden layers and on
  arXiv  Detail & Related papers  (2023-05-04T18:28:18Z)
• Flatter, faster: scaling momentum for optimal speedup of SGD [0.0]
  We study training dynamics arising from interplay between gradient descent (SGD) and label noise and momentum in the training of neural networks. We find that scaling the momentum hyper parameter $1-NISTbeta$ with the learning rate to the power of $2/3$ maximally accelerates training, without sacrificing generalization.
  arXiv  Detail & Related papers  (2022-10-28T20:41:48Z)
• Multi-scale Feature Learning Dynamics: Insights for Double Descent [71.91871020059857]
  We study the phenomenon of "double descent" of the generalization error. We find that double descent can be attributed to distinct features being learned at different scales.
  arXiv  Detail & Related papers  (2021-12-06T18:17:08Z)
• Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization [118.50301177912381]
  We show that Adam can converge to different solutions of the objective with provably different errors, even with weight decay globalization. We show that if convex, and the weight decay regularization is employed, any optimization algorithms including Adam will converge to the same solution.
  arXiv  Detail & Related papers  (2021-08-25T17:58:21Z)
• Neurons learn slower than they think [0.0]
  This study introduces textitdifferential capability into an optimization process. It measures whether the test accuracy increases as fast as a model approaches the decision boundary in a classification problem.
  arXiv  Detail & Related papers  (2021-04-02T09:09:52Z)
• Adaptive Gradient Method with Resilience and Momentum [120.83046824742455]
  We propose an Adaptive Gradient Method with Resilience and Momentum (AdaRem) AdaRem adjusts the parameter-wise learning rate according to whether the direction of one parameter changes in the past is aligned with the direction of the current gradient. Our method outperforms previous adaptive learning rate-based algorithms in terms of the training speed and the test error.
  arXiv  Detail & Related papers  (2020-10-21T14:49:00Z)
• Multiplicative noise and heavy tails in stochastic optimization [62.993432503309485]
  empirical optimization is central to modern machine learning, but its role in its success is still unclear. We show that it commonly arises in parameters of discrete multiplicative noise due to variance. A detailed analysis is conducted in which we describe on key factors, including recent step size, and data, all exhibit similar results on state-of-the-art neural network models.
  arXiv  Detail & Related papers  (2020-06-11T09:58:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.