Overshoot: Taking advantage of future gradients in momentum-based stochastic optimization

• URL: http://arxiv.org/abs/2501.09556v1
• Date: Thu, 16 Jan 2025 14:18:10 GMT
• Title: Overshoot: Taking advantage of future gradients in momentum-based stochastic optimization
• Authors: Jakub Kopal, Michal Gregor, Santiago de Leon-Martinez, Jakub Simko,
• Abstract summary: Overshoot is a momentum-based descent optimization method designed to enhance performance beyond standard and Nesterov's momentum. Overshoot consistently outperforms both standard and Nesterov's momentum across a wide range of tasks.
• Score: 1.4303041760959478
• License:
• Abstract: Overshoot is a novel, momentum-based stochastic gradient descent optimization method designed to enhance performance beyond standard and Nesterov's momentum. In conventional momentum methods, gradients from previous steps are aggregated with the gradient at current model weights before taking a step and updating the model. Rather than calculating gradient at the current model weights, Overshoot calculates the gradient at model weights shifted in the direction of the current momentum. This sacrifices the immediate benefit of using the gradient w.r.t. the exact model weights now, in favor of evaluating at a point, which will likely be more relevant for future updates. We show that incorporating this principle into momentum-based optimizers (SGD with momentum and Adam) results in faster convergence (saving on average at least 15% of steps). Overshoot consistently outperforms both standard and Nesterov's momentum across a wide range of tasks and integrates into popular momentum-based optimizers with zero memory and small computational overhead.

Related papers

• The Marginal Value of Momentum for Small Learning Rate SGD [20.606430391298815]
  Momentum is known to accelerate the convergence of gradient descent in strongly convex settings without gradient noise regimes. Experiments show that momentum indeed has limited benefits for both optimization and generalization in practical training where the optimal learning rate is not very large.
  arXiv  Detail & Related papers  (2023-07-27T21:01:26Z)
• Hidden Markov Models with Momentum [6.48893856598641]
  We experiment with adding momentum to the Baum-Welch expectation-maximization algorithm for training Hidden Markov Models. Our experiments indicate that adding momentum to Baum-Welch can reduce the number of iterations required for initial convergence. However, momentum does not seem to improve the final model performance at a high number of iterations.
  arXiv  Detail & Related papers  (2022-06-08T15:49:43Z)
• Deep Equilibrium Optical Flow Estimation [80.80992684796566]
  Recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms. These RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. We propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer.
  arXiv  Detail & Related papers  (2022-04-18T17:53:44Z)
• On Training Implicit Models [75.20173180996501]
  We propose a novel gradient estimate for implicit models, named phantom gradient, that forgoes the costly computation of the exact gradient. Experiments on large-scale tasks demonstrate that these lightweight phantom gradients significantly accelerate the backward passes in training implicit models by roughly 1.7 times.
  arXiv  Detail & Related papers  (2021-11-09T14:40:24Z)
• Accelerate Distributed Stochastic Descent for Nonconvex Optimization with Momentum [12.324457683544132]
  We propose a momentum method for such model averaging approaches. We analyze the convergence and scaling properties of such momentum methods. Our experimental results show that block momentum not only accelerates training, but also achieves better results.
  arXiv  Detail & Related papers  (2021-10-01T19:23:18Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)
• Adapting Stepsizes by Momentumized Gradients Improves Optimization and Generalization [89.66571637204012]
  textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing. textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing. textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing.
  arXiv  Detail & Related papers  (2021-06-22T03:13:23Z)
• Escaping Saddle Points Faster with Stochastic Momentum [9.485782209646445]
  In deep networks, momentum appears to significantly improve convergence time. We show that momentum improves deep training because it modifies SGD to escape points faster. We also show how to choose the ideal momentum parameter.
  arXiv  Detail & Related papers  (2021-06-05T23:34:02Z)
• AdamP: Slowing Down the Slowdown for Momentum Optimizers on Scale-invariant Weights [53.8489656709356]
  Normalization techniques are a boon for modern deep learning. It is often overlooked, however, that the additional introduction of momentum results in a rapid reduction in effective step sizes for scale-invariant weights. In this paper, we verify that the widely-adopted combination of the two ingredients lead to the premature decay of effective step sizes and sub-optimal model performances.
  arXiv  Detail & Related papers  (2020-06-15T08:35:15Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• A New Accelerated Stochastic Gradient Method with Momentum [4.967897656554012]
  gradient descent with momentum (Sgdm) use weights that decay exponentially with the iteration times to generate an momentum term. We provide theoretical convergence properties analyses for our method, which show both the exponentially decay weights and our inverse proportionally decay weights can limit the variance of the moving direction of parameters to be optimized to a region.
  arXiv  Detail & Related papers  (2020-05-31T03:04:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.