Flatland: The Adventures of Gradient Descent with Large Step Sizes

• URL: http://arxiv.org/abs/2606.06722v1
• Date: Thu, 04 Jun 2026 21:14:07 GMT
• Title: Flatland: The Adventures of Gradient Descent with Large Step Sizes
• Abstract summary: We provide a unifying definition of "large" step sizes for gradient descent training.<n>We show that large step sizes operate at the edge of stability (EoS) right from the start of the training.<n>Contrary to expectation, we find that encountering globally-flat regions too early in the training may both slow down convergence and jeopardize the generalization ability of the network.
• Score: 11.094067941432144
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The training of neural networks often entails objective functions that are not globally $L$-smooth. For these functions, it is both theoretically and practically difficult to reply to the question: what is the largest possible step size that ensures the convergence of gradient descent (GD)? We address this longstanding open question in deep learning by providing a unifying definition of "large" step sizes that requires only local Lipschitz (or even Hölder) continuity of the gradient. We design first-order adaptive methods that provably yield large step sizes and show that they operate at the edge of stability (EoS) right from the start of the training. In particular, the loss decreases nonmonotonically and the product between the step size and sharpness, i.e., the largest eigenvalue of the Hessian, stays above the EoS threshold of 2 throughout training. Using our method, we are also able to minimize the sharpness all the way down to its global minimum. Contrary to expectation, we find that encountering globally-flat regions too early in the training may both slow down convergence and jeopardize the generalization ability of the network. Exploiting a self-stabilization argument, we allow GD to enter slightly sharper valleys and turn unsuccessful training runs into very successful ones.

Related papers

• Non-Singularity of the Gradient Descent map for Neural Networks with Piecewise Analytic Activations [53.348574336527854]
  We investigate the neural network map as a function on the space of weights and biases.<n>We prove, for the first time, the non-singularity of the gradient descent (GD) map on the loss landscape of realistic neural network architectures.
  arXiv  Detail & Related papers  (2025-10-28T14:34:33Z)
• First-ish Order Methods: Hessian-aware Scalings of Gradient Descent [11.125968799758436]
  A key limitation of gradient descent is its lack of natural scaling.<n>By accounting for curvature, our adaptive, Hessian-aware scaling method ensures a local unit step size.<n>We show that our method converges globally under a significantly weaker version of the standard Lipschitz assumption.
  arXiv  Detail & Related papers  (2025-02-06T01:22:23Z)
• Gradient Descent Converges Linearly to Flatter Minima than Gradient Flow in Shallow Linear Networks [0.0]
  We study the gradient descent dynamics of a depth-2 linear neural network with a single input and output.<n>We show that GD converges at an explicit linear rate to a global minimum of the training loss, even with a large stepsize.
  arXiv  Detail & Related papers  (2025-01-15T20:43:36Z)
• On the Convergence of Gradient Descent for Large Learning Rates [55.33626480243135]
  We show that convergence is impossible when a fixed step size is used.<n>We provide a proof of this in the case of linear neural networks with a squared loss.<n>We also prove the impossibility of convergence for more general losses without requiring strong assumptions such as Lipschitz continuity for the gradient.
  arXiv  Detail & Related papers  (2024-02-20T16:01:42Z)
• The Effect of SGD Batch Size on Autoencoder Learning: Sparsity, Sharpness, and Feature Learning [14.004531386769328]
  We investigate the dynamics of gradient descent (SGD) when a single-neuron autoencoder is used. For any batch size smaller than the number of samples, SGD finds a global minimum which is sparse and nearly strictly to its randomness.
  arXiv  Detail & Related papers  (2023-08-06T21:54:07Z)
• Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data [63.34506218832164]
  In this work, we investigate the implicit bias of gradient flow and gradient descent in two-layer fully-connected neural networks with ReLU activations. For gradient flow, we leverage recent work on the implicit bias for homogeneous neural networks to show that leakyally, gradient flow produces a neural network with rank at most two. For gradient descent, provided the random variance is small enough, we show that a single step of gradient descent suffices to drastically reduce the rank of the network, and that the rank remains small throughout training.
  arXiv  Detail & Related papers  (2022-10-13T15:09:54Z)
• On Avoiding Local Minima Using Gradient Descent With Large Learning Rates [48.42061343682201]
  We argue that a complete understanding of the mechanics leading to descent (GD)'s success may indeed require considering effects of using a large step size. We show that GD with large step size follows a different trajectory than GD with a small step size, leading to convergence to the global minimum. We also demonstrate the difference in trajectories for small and large learning rates when GD is applied on a neural network.
  arXiv  Detail & Related papers  (2022-05-30T14:35:32Z)
• Adapting Stepsizes by Momentumized Gradients Improves Optimization and Generalization [89.66571637204012]
  textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing. textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing. textscAdaMomentum on vision, and achieves state-the-art results consistently on other tasks including language processing.
  arXiv  Detail & Related papers  (2021-06-22T03:13:23Z)
• Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracy [71.25689267025244]
  We show how the transition is controlled by the relationship between the scale and how accurately we minimize the training loss. Our results indicate that some limit behaviors of gradient descent only kick in at ridiculous training accuracies.
  arXiv  Detail & Related papers  (2020-07-13T23:49:53Z)
• Can Shallow Neural Networks Beat the Curse of Dimensionality? A mean field training perspective [8.160343645537106]
  gradient descent training of a two-layer neural network on empirical or population risk may not decrease population risk at an order faster than $t-4/(d-2)$ under mean field scaling. We present numerical evidence that gradient descent training with general Lipschitz target functions becomes slower and slower as the dimension increases, but converges at approximately the same rate in all dimensions when the target function lies in the natural function space for two-layer ReLU networks.
  arXiv  Detail & Related papers  (2020-05-21T17:50:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.