A Bifurcation Theory Framework for Gradient Descent on the Edge of Stability

• URL: http://arxiv.org/abs/2606.15551v1
• Date: Sun, 14 Jun 2026 02:39:54 GMT
• Title: A Bifurcation Theory Framework for Gradient Descent on the Edge of Stability
• Abstract summary: We develop a bifurcation theory framework for gradient descent on the edge of stability.<n>We show that stable EoS training arises from a flip bifurcation in the normal direction.
• Score: 3.4773470589069473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Edge of Stability (EoS) phenomenon, where gradient descent operates with sharpness exceeding the classical convergence threshold yet the loss decreases over long timescales, is ubiquitous in modern deep learning but remains poorly understood in realistic settings. Prior rigorous analyses have been largely confined to scalar or low-dimensional losses with specific structural forms. In this work, we develop a bifurcation theory framework for gradient descent on the edge of stability that applies directly to overparameterized neural networks. By decomposing the training dynamics into components normal and tangent to the manifold of minimizers, we show that stable EoS training arises from a flip bifurcation in the normal direction, governed by the sign of the first Lyapunov coefficient, while the tangent dynamics drift toward regions of decreasing sharpness. Under mild spectral and geometric assumptions on the loss landscape, we prove convergence to the minimizing manifold when training at the EoS threshold. As a corollary, we recover and unify prior results: we show that the product-stability condition of Gan (2026) is an instance of our framework.

Related papers

• Across the Loss Landscape with Progressive Growth [51.366966420881646]
  We show how grow-and-optimize strategies to train toward flatter regions by viewing growth as progressive relaxation.<n>We empirically validate these predictions in controlled toy basins and in a realistic ResNet/CI-100 setting.<n>Although progressive subspace growth reliably produces flatter solutions, curvature reductions do not universally translate into improved test performance, highlighting subtleties in the flatness-generalization connection.
  arXiv  Detail & Related papers  (2026-08-25T13:50:48Z)
• SDS-LoRA: Overcoming Anisotropic Gradient Scaling in Low-Rank Adaptation [64.33799467286265]
  Low-Rank Adaptation (LoRA) enables efficient adaptation of large pre-trained models to downstream tasks.<n>We show that when a full fine-tuning gradient is backpropagated to the low-rank matrices, it undergoes anisotropic scaling driven by their singular values.<n>We propose a new low-rank parameterization, SDS-LoRA, which structurally decouples singular values from the backward pass.
  arXiv  Detail & Related papers  (2026-06-15T09:27:50Z)
• Product-Stability: Provable Convergence for Gradient Descent on the Edge of Stability [3.4773470589069473]
  We introduce and study a structural property of loss functions that we term product-stability.<n>We show that for losses with product-stable minima, gradient descent applied to objectives of the form $(x,y) mapsto l(xy)$ can provably converge to the local minimum even when training in the Edge of Stability regime.
  arXiv  Detail & Related papers  (2026-04-03T02:35:05Z)
• Training Instabilities Induce Flatness Bias in Gradient Descent [6.628332915214955]
  Modern deep networks often achieve their best performance beyond a stability threshold.<n>We show that training instabilities induce an implicit bias in GD, driving parameters toward flatter regions of the loss landscape.<n>We also show that restoring instabilities in Adam further improves generalization.
  arXiv  Detail & Related papers  (2025-11-16T11:26:25Z)
• Description of the Training Process of Neural Networks via Ergodic Theorem : Ghost nodes [3.637162892228131]
  We present a unified framework for understanding and accelerating deep neural networks via training gradient descent (SGD)<n>We introduce a practical diagnostic, the running estimate of the largest Lyapunov exponent, which distinguishes genuine convergence toward stablers.<n>We propose a ghost category extension for standard classifiers that adds auxiliary ghost output nodes so the model gains extra descent directions.
  arXiv  Detail & Related papers  (2025-07-01T17:54:35Z)
• On the Convergence of Gradient Descent for Large Learning Rates [55.33626480243135]
  We show that convergence is impossible when a fixed step size is used.<n>We provide a proof of this in the case of linear neural networks with a squared loss.<n>We also prove the impossibility of convergence for more general losses without requiring strong assumptions such as Lipschitz continuity for the gradient.
  arXiv  Detail & Related papers  (2024-02-20T16:01:42Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• Universal Sharpness Dynamics in Neural Network Training: Fixed Point Analysis, Edge of Stability, and Route to Chaos [5.854190253899593]
  In gradient descent dynamics of neural networks, the top eigenvalue of the loss Hessian (sharpness) displays a variety of robust phenomena throughout training.<n>We demonstrate that a simple $2$-layer linear network (UV model) trained on a single training example exhibits all of the essential sharpness phenomenology observed in real-world scenarios.
  arXiv  Detail & Related papers  (2023-11-03T17:59:40Z)
• Trajectory Alignment: Understanding the Edge of Stability Phenomenon via Bifurcation Theory [14.141453107129403]
  We study the evolution of the largest eigenvalue of the loss Hessian, also known as sharpness, along the gradient descent trajectory. The sharpness increases at the early phase of training, and eventually saturates close to the threshold of $2 / text(step size)$.
  arXiv  Detail & Related papers  (2023-07-09T15:16:45Z)
• Beyond the Edge of Stability via Two-step Gradient Updates [49.03389279816152]
  Gradient Descent (GD) is a powerful workhorse of modern machine learning. GD's ability to find local minimisers is only guaranteed for losses with Lipschitz gradients. This work focuses on simple, yet representative, learning problems via analysis of two-step gradient updates.
  arXiv  Detail & Related papers  (2022-06-08T21:32:50Z)
• Gradient Starvation: A Learning Proclivity in Neural Networks [97.02382916372594]
  Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks.
  arXiv  Detail & Related papers  (2020-11-18T18:52:08Z)
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks [64.7563588124004]
  We argue for the existence of the "break-even" point on this trajectory. We show that using a large learning rate in the initial phase of training reduces the variance of the gradient. We also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers.
  arXiv  Detail & Related papers  (2020-02-21T22:55:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.