Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima

• URL: http://arxiv.org/abs/2607.08380v1
• Date: Thu, 09 Jul 2026 11:58:44 GMT
• Title: Dynamics of Gradient Descent with Large Step Size Near a Manifold of Flat Minima
• Abstract summary: An important quantity in the theory of gradient descent (GD) is the emphsharpness, defined as the largest eigenvalue of the objective Hessian.<n>Recent work bridges this gap in the setting of overparametrised least-squares with a emphsingle scalar output<n>We show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results.
• Score: 39.50927136881194
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: An important quantity in the theory of gradient descent (GD) is the \emph{sharpness}, defined as the largest eigenvalue of the objective Hessian. Classical analyses typically require the step size to be uniformly smaller than twice the reciprocal of the sharpness, but this condition is frequently violated in the training of deep neural networks. Recent work bridges this gap in the setting of overparametrised least-squares with a \emph{single scalar output}, providing a normal form for large-step GD in a neighbourhood of an \emph{isolated} flat minimum and establishing three corresponding convergence results. In this paper, we extend this theory in two directions: (1) to overparametrised least-squares with \emph{vector-valued outputs} (including regression with arbitrarily many observations), and (2) to a neighbourhood of a \emph{manifold} of flat minima (which we show is essential for applications such as matrix factorisation). We generalise both the normal form and all three convergence theorems of \cite{macdonaldeos} to this broader setting, overcoming several technical challenges, including the solution of a singular partial differential equation via a novel method that may be of independent interest. We further show that our framework applies to deep matrix factorisation under mild assumptions, yielding several new structural results. In particular, we prove that the set of flat minima forms a fibre bundle over a product of spheres, and that the sharpness is Morse-Bott along this manifold.

Related papers

• Dimension Reduction for Curves: Simplified and Generalized [56.23281156053955]
  We revisit random projections for reducing the dimension of high-dimensional polygonal curves.<n>We give a simplified proof of the known $O(varepsilon-2log(nm))$ bound on the target dimension of a random projection.<n>Our proof is based on the concept of sparse oblivious subspace embeddings.
  arXiv  Detail & Related papers  (2026-07-03T08:48:35Z)
• Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks [3.011579195399507]
  We analytically derive the gradient of the WS upper bound to characterize directions leading to flat minima.<n> Experiments demonstrate that HSR Regularization narrows the Hessian eigenvalue spectrum, avoids sharp minima and saddle points, and promotes convergence to flat minima.
  arXiv  Detail & Related papers  (2026-06-27T00:47:03Z)
• Fitting Unknown Number of Hyperplanes with Manifold Optimization [57.48093263119306]
  Fitting unknown number of linearplanes to data is a fundamental yet challenging problem in machine learning.<n>Existing approaches often struggle with optimum optima or lack geometric consistency.
  arXiv  Detail & Related papers  (2026-05-27T14:02:20Z)
• The Magnitude of Dominated Sets: A Pareto Compliant Indicator Grounded in Metric Geometry [0.0]
  Magnitude is a notion of size or point content for compact metric spaces.<n>For dominated regions in the (ell_1) box setting, magnitude is close to hypervolume.<n>Unlike hypervolume, magnitude assigns positive value to boundary points sharing one or more coordinates with the anchor point.
  arXiv  Detail & Related papers  (2026-04-20T12:10:12Z)
• Stable Minima of ReLU Neural Networks Suffer from the Curse of Dimensionality: The Neural Shattering Phenomenon [25.998397575754865]
  We study the implicit bias of flatness / low (loss) curvature and its effects on generalization in ReLU networks.<n>We show that while flatness does imply generalization, the resulting rates of convergence necessarily deteriorate exponentially as the input dimension grows.
  arXiv  Detail & Related papers  (2025-06-25T19:10:03Z)
• Geometric Meta-Learning via Coupled Ricci Flow: Unifying Knowledge Representation and Quantum Entanglement [7.410691988131121]
  This paper establishes a unified framework integrating geometric flows with deep learning through three fundamental innovations.<n>First, we propose a thermodynamically coupled Ricci flow that dynamically adapts parameter space geometry to loss landscape topology.<n>Second, we derive explicit phase transition thresholds and critical learning rates through curvature blowup analysis.<n>Third, we establish an AdS/CFT-type holographic duality (Theoremrefthm:ads) between neural networks and conformal field theories.
  arXiv  Detail & Related papers  (2025-03-25T17:32:31Z)
• Pushing the Limits of Large Language Model Quantization via the Linearity Theorem [71.3332971315821]
  We present a "line theoremarity" establishing a direct relationship between the layer-wise $ell$ reconstruction error and the model perplexity increase due to quantization. This insight enables two novel applications: (1) a simple data-free LLM quantization method using Hadamard rotations and MSE-optimal grids, dubbed HIGGS, and (2) an optimal solution to the problem of finding non-uniform per-layer quantization levels.
  arXiv  Detail & Related papers  (2024-11-26T15:35:44Z)
• The Dynamics of Sharpness-Aware Minimization: Bouncing Across Ravines and Drifting Towards Wide Minima [41.961056785108845]
  We consider Sharpness-Aware Minimization, a gradient-based optimization method for deep networks. We show that when SAM is applied with a convex quadratic objective, it converges to a cycle that oscillates between either side of the minimum in the direction with the largest curvature. In the non-quadratic case, we show that such oscillations effectively perform gradient descent, with a smaller step-size, on the spectral norm of the Hessian.
  arXiv  Detail & Related papers  (2022-10-04T10:34:37Z)
• Lifting the Convex Conjugate in Lagrangian Relaxations: A Tractable Approach for Continuous Markov Random Fields [53.31927549039624]
  We show that a piecewise discretization preserves better contrast from existing discretization problems. We apply this theory to the problem of matching two images.
  arXiv  Detail & Related papers  (2021-07-13T12:31:06Z)
• Differentiating through the Fr\'echet Mean [51.32291896926807]
  Fr'echet mean is a generalization of the Euclidean mean. We show how to differentiate through the Fr'echet mean for arbitrary Riemannian manifold. This fully integrates the Fr'echet mean into the hyperbolic neural network pipeline.
  arXiv  Detail & Related papers  (2020-02-29T19:49:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.