Sharpness of Minima in Deep Matrix Factorization: Exact Expressions

• URL: http://arxiv.org/abs/2509.25783v2
• Date: Thu, 02 Oct 2025 19:50:49 GMT
• Title: Sharpness of Minima in Deep Matrix Factorization: Exact Expressions
• Abstract summary: We present the first exact expression for the Hessian of the squared-error loss at any minimizer.<n>We investigate an escape phenomenon observed during gradient-based training.
• Score: 5.736588561666141
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Understanding the geometry of the loss landscape near a minimum is key to explaining the implicit bias of gradient-based methods in non-convex optimization problems such as deep neural network training and deep matrix factorization. A central quantity to characterize this geometry is the maximum eigenvalue of the Hessian of the loss, which measures the sharpness of the landscape. Currently, its precise role has been obfuscated because no exact expressions for this sharpness measure were known in general settings. In this paper, we present the first exact expression for the maximum eigenvalue of the Hessian of the squared-error loss at any minimizer in general overparameterized deep matrix factorization (i.e., deep linear neural network training) problems, resolving an open question posed by Mulayoff & Michaeli (2020). To complement our theory, we empirically investigate an escape phenomenon observed during gradient-based training near a minimum that crucially relies on our exact expression of the sharpness.

Related papers

• Closed-Form Steepest Descent Direction toward Flat Minima: Reducing Upper Bounds on the Loss Hessian Eigenspectrum in Neural Networks [3.011579195399507]
  We analytically derive the gradient of the WS upper bound to characterize directions leading to flat minima.<n> Experiments demonstrate that HSR Regularization narrows the Hessian eigenvalue spectrum, avoids sharp minima and saddle points, and promotes convergence to flat minima.
  arXiv  Detail & Related papers  (2026-06-27T00:47:03Z)
• Wolkowicz-Styan Upper Bound on the Hessian Eigenspectrum for Cross-Entropy Loss in Nonlinear Smooth Neural Networks [3.011579195399507]
  Relationship between loss geometry and generalization is still not well understood.<n>This study focuses on nonlinear, smooth multilayer neural networks and derives a closed-form upper bound for the maximum eigenvalue of the Hessian.<n>The derived upper bound is expressed as a function of the affine transformation parameters, hidden layer dimensions, and the degree of computationity among the training samples.
  arXiv  Detail & Related papers  (2026-04-11T13:16:29Z)
• On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness [5.736588561666141]
  Hessian spectrum is constant across all minimizers of regularized deep scalar factorization problems with squared-error loss.<n>If the target matrix does not belong to the Lebesgue measure-zero set, then the Frobenius norm of each layer is constant across all minimizers.
  arXiv  Detail & Related papers  (2026-03-28T01:22:01Z)
• Gradient Flow Equations for Deep Linear Neural Networks: A Survey from a Network Perspective [0.0]
  The paper surveys recent progresses in understanding the dynamics and loss landscape of the gradient flow equations associated to deep linear neural networks.<n>The loss landscape is characterized by infinitely many global minima and saddle points, both strict and nonstrict, but lacks local minima and maxima.<n>The adjacency matrix representation we use in the paper allows to highlight the existence of a quotient space structure.
  arXiv  Detail & Related papers  (2025-11-13T14:39:32Z)
• A Universal Class of Sharpness-Aware Minimization Algorithms [57.29207151446387]
  We introduce a new class of sharpness measures, leading to new sharpness-aware objective functions. We prove that these measures are textitly expressive, allowing any function of the training loss Hessian matrix to be represented by appropriate hyper and determinants.
  arXiv  Detail & Related papers  (2024-06-06T01:52:09Z)
• Deep linear networks for regression are implicitly regularized towards flat minima [4.806579822134391]
  Minimizers can have arbitrarily large sharpness, but not an arbitrarily small one. We show a lower bound on the sharpness of minimizers, which grows linearly with depth. We show an implicit regularization towards flat minima: the sharpness of the minimizer is no more than a constant times the lower bound.
  arXiv  Detail & Related papers  (2024-05-22T08:58:51Z)
• A Mean-Field Analysis of Neural Stochastic Gradient Descent-Ascent for Functional Minimax Optimization [90.87444114491116]
  This paper studies minimax optimization problems defined over infinite-dimensional function classes of overparametricized two-layer neural networks. We address (i) the convergence of the gradient descent-ascent algorithm and (ii) the representation learning of the neural networks. Results show that the feature representation induced by the neural networks is allowed to deviate from the initial one by the magnitude of $O(alpha-1)$, measured in terms of the Wasserstein distance.
  arXiv  Detail & Related papers  (2024-04-18T16:46:08Z)
• Understanding the training of infinitely deep and wide ResNets with Conditional Optimal Transport [26.47265060394168]
  We show that the gradient flow for deep neural networks converges arbitrarily at a distance ofr.<n>This is done by relying on the theory of gradient distance of finite width in spaces.
  arXiv  Detail & Related papers  (2024-03-19T16:34:31Z)
• The Inductive Bias of Flatness Regularization for Deep Matrix Factorization [58.851514333119255]
  This work takes the first step toward understanding the inductive bias of the minimum trace of the Hessian solutions in deep linear networks. We show that for all depth greater than one, with the standard Isometry Property (RIP) on the measurements, minimizing the trace of Hessian is approximately equivalent to minimizing the Schatten 1-norm of the corresponding end-to-end matrix parameters.
  arXiv  Detail & Related papers  (2023-06-22T23:14:57Z)
• Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Data [63.34506218832164]
  In this work, we investigate the implicit bias of gradient flow and gradient descent in two-layer fully-connected neural networks with ReLU activations. For gradient flow, we leverage recent work on the implicit bias for homogeneous neural networks to show that leakyally, gradient flow produces a neural network with rank at most two. For gradient descent, provided the random variance is small enough, we show that a single step of gradient descent suffices to drastically reduce the rank of the network, and that the rank remains small throughout training.
  arXiv  Detail & Related papers  (2022-10-13T15:09:54Z)
• Linear Adversarial Concept Erasure [108.37226654006153]
  We formulate the problem of identifying and erasing a linear subspace that corresponds to a given concept.<n>We show that the method is highly expressive, effectively mitigating bias in deep nonlinear classifiers while maintaining tractability and interpretability.
  arXiv  Detail & Related papers  (2022-01-28T13:00:17Z)
• Deep learning: a statistical viewpoint [120.94133818355645]
  Deep learning has revealed some major surprises from a theoretical perspective. In particular, simple gradient methods easily find near-perfect solutions to non-optimal training problems. We conjecture that specific principles underlie these phenomena.
  arXiv  Detail & Related papers  (2021-03-16T16:26:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.