On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness

• URL: http://arxiv.org/abs/2603.27072v1
• Date: Sat, 28 Mar 2026 01:22:01 GMT
• Title: On the Loss Landscape Geometry of Regularized Deep Matrix Factorization: Uniqueness and Sharpness
• Abstract summary: Hessian spectrum is constant across all minimizers of regularized deep scalar factorization problems with squared-error loss.<n>If the target matrix does not belong to the Lebesgue measure-zero set, then the Frobenius norm of each layer is constant across all minimizers.
• Score: 5.736588561666141
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Weight decay is ubiquitous in training deep neural network architectures. Its empirical success is often attributed to capacity control; nonetheless, our theoretical understanding of its effect on the loss landscape and the set of minimizers remains limited. In this paper, we show that $\ell^2$-regularized deep matrix factorization/deep linear network training problems with squared-error loss admit a unique end-to-end minimizer for all target matrices subject to factorization, except for a set of Lebesgue measure zero formed by the depth and the regularization parameter. This observation reveals fundamental properties of the loss landscape of regularized deep matrix factorization problems: the Hessian spectrum is constant across all minimizers of the regularized deep scalar factorization problem with squared-error loss. Moreover, we show that, in regularized deep matrix factorization problems with squared-error loss, if the target matrix does not belong to the Lebesgue measure-zero set, then the Frobenius norm of each layer is constant across all minimizers. This, in turn, yields a global lower bound on the trace of the Hessian evaluated at any minimizer of the regularized deep matrix factorization problem. Furthermore, we establish a critical threshold for the regularization parameter above which the unique end-to-end minimizer collapses to zero.

Related papers

• The Hidden Cost of Approximation in Online Mirror Descent [56.99972253009168]
  Online mirror descent (OMD) is a fundamental algorithmic paradigm that underlies many algorithms in optimization, machine learning and sequential decision-making.<n>In this work we initiate a systematic study into inexact OMD, and uncover an intricate relation between regularizer smoothness and robustness to approximation errors.
  arXiv  Detail & Related papers  (2025-11-27T10:09:07Z)
• Sharpness of Minima in Deep Matrix Factorization: Exact Expressions [5.736588561666141]
  We present the first exact expression for the Hessian of the squared-error loss at any minimizer.<n>We investigate an escape phenomenon observed during gradient-based training.
  arXiv  Detail & Related papers  (2025-09-30T04:50:28Z)
• Spectral Analysis of the Weighted Frobenius Objective [2.7857707184602165]
  We analyze a weighted Frobenius loss for approxingimating symmetric definite matrices in the context of preconditioning iterative solvers.<n>Our analysis reveals that each eigenmode is scaled by the corresponding square of its eigenvalue, and that, under a fixed error budget, the loss is minimized only when the error is confined to the direction of the largest eigenvalue.
  arXiv  Detail & Related papers  (2025-09-20T19:15:24Z)
• A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorization [4.6485895241404585]
  We conduct a comprehensive study of the loss landscape of the regularized DMF problem.<n>We derive a necessary and sufficient condition under which every critical point is either a local minimizer or a strict saddle point.<n>This provides insights into why gradient-based methods almost always converge to a local minimizer of the regularized DMF problem.
  arXiv  Detail & Related papers  (2025-06-25T11:51:41Z)
• A Universal Class of Sharpness-Aware Minimization Algorithms [57.29207151446387]
  We introduce a new class of sharpness measures, leading to new sharpness-aware objective functions. We prove that these measures are textitly expressive, allowing any function of the training loss Hessian matrix to be represented by appropriate hyper and determinants.
  arXiv  Detail & Related papers  (2024-06-06T01:52:09Z)
• Entrywise error bounds for low-rank approximations of kernel matrices [55.524284152242096]
  We derive entrywise error bounds for low-rank approximations of kernel matrices obtained using the truncated eigen-decomposition. A key technical innovation is a delocalisation result for the eigenvectors of the kernel matrix corresponding to small eigenvalues. We validate our theory with an empirical study of a collection of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-05-23T12:26:25Z)
• The Inductive Bias of Flatness Regularization for Deep Matrix Factorization [58.851514333119255]
  This work takes the first step toward understanding the inductive bias of the minimum trace of the Hessian solutions in deep linear networks. We show that for all depth greater than one, with the standard Isometry Property (RIP) on the measurements, minimizing the trace of Hessian is approximately equivalent to minimizing the Schatten 1-norm of the corresponding end-to-end matrix parameters.
  arXiv  Detail & Related papers  (2023-06-22T23:14:57Z)
• The loss landscape of deep linear neural networks: a second-order analysis [9.85879905918703]
  We study the optimization landscape of deep linear neural networks with the square loss. We characterize, among all critical points, which are global minimizers, strict saddle points, and non-strict saddle points.
  arXiv  Detail & Related papers  (2021-07-28T11:33:18Z)
• Understanding Implicit Regularization in Over-Parameterized Single Index Model [55.41685740015095]
  We design regularization-free algorithms for the high-dimensional single index model. We provide theoretical guarantees for the induced implicit regularization phenomenon.
  arXiv  Detail & Related papers  (2020-07-16T13:27:47Z)
• Relative Error Bound Analysis for Nuclear Norm Regularized Matrix Completion [101.83262280224729]
  We develop a relative error bound for nuclear norm regularized matrix completion. We derive a relative upper bound for recovering the best low-rank approximation of the unknown matrix.
  arXiv  Detail & Related papers  (2015-04-26T13:12:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.