Fisher-Geometric Sharpness and the Implicit Bias of SGD toward Flat Minima

• URL: http://arxiv.org/abs/2606.20469v1
• Date: Thu, 18 Jun 2026 16:48:18 GMT
• Title: Fisher-Geometric Sharpness and the Implicit Bias of SGD toward Flat Minima
• Abstract summary: gradient descent (SGD) implicitly favors flat minima.<n>We prove that it is invariant under smooth, function-preserving reparametrizations.<n>Experiments on MNIST and CIFAR-10 confirm that SR reliably tracks generalization in ways that Euclidean sharpness does not.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A widely held intuition in deep learning is that stochastic gradient descent (SGD) implicitly favors flat minima and that flat minima generalize better, but standard Euclidean measures of flatness such as the trace or maximum eigenvalue of the loss Hessian are not invariant under reparametrizations that preserve the network function, which undermines the theoretical foundations of this narrative. In this study we resolve this issue by grounding flatness in the Riemannian geometry of the statistical manifold induced by the Fisher Information Matrix (FIM). We define Riemannian sharpness mathematically and prove that it is invariant under smooth, function-preserving reparametrizations, which directly addresses the critique of Dinh et al. in the paper ``Sharp minima can generalize for deep nets''.We note that this invariance is a property of the true FIM; the diagonal empirical estimator used in practice (and in all experiments below) inherits invariance only approximately, and exact invariance under arbitrary reparametrizations would require structured estimators such as K-FAC. We formalize the gradient noise of mini-batch SGD as having a covariance structure proportional to the FIM, derive the stationary distribution of the resulting stochastic differential equation, and then show that the probability mass is exponentially concentrated at Riemannian-flat minima. A PAC-Bayes generalization bound controlled explicitly by SR formally links this geometric bias to test performance. Our experiments on MNIST and CIFAR-10 confirm that SR reliably tracks generalization in ways that Euclidean sharpness does not, and that its scaling with $η/B$ matches the theoretical predictions. Together these results provide a rigorous, reparametrization-invariant account of why flat minima generalize.

Related papers

• Tubular Riemannian Laplace Approximations for Bayesian Neural Networks [0.0]
  Laplace approximations are among the simplest and most practical methods for approximate Bayesian inference in neural networks.<n>Recent work has proposed geometric Gaussian approximations to adapt to this structure.<n>We introduce the Tubular Riemannian Laplace (TRL) approximation.
  arXiv  Detail & Related papers  (2025-12-30T17:50:55Z)
• A Geometric Unification of Distributionally Robust Covariance Estimators: Shrinking the Spectrum by Inflating the Ambiguity Set [20.166217494056916]
  We propose a principled approach to construct covariance estimators without imposing restrictive assumptions.<n>We show that our robust estimators are efficiently computable and consistent.<n> Numerical experiments based on synthetic and real data show that our robust estimators are competitive with state-of-the-art estimators.
  arXiv  Detail & Related papers  (2024-05-30T15:01:18Z)
• Intrinsic Bayesian Cramér-Rao Bound with an Application to Covariance Matrix Estimation [49.67011673289242]
  This paper presents a new performance bound for estimation problems where the parameter to estimate lies in a smooth manifold. It induces a geometry for the parameter manifold, as well as an intrinsic notion of the estimation error measure.
  arXiv  Detail & Related papers  (2023-11-08T15:17:13Z)
• Spectrum-Aware Debiasing: A Modern Inference Framework with Applications to Principal Components Regression [1.342834401139078]
  We introduce SpectrumAware Debiasing, a novel method for high-dimensional regression. Our approach applies to problems with structured, heavy tails, and low-rank structures. We demonstrate our method through simulated and real data experiments.
  arXiv  Detail & Related papers  (2023-09-14T15:58:30Z)
• Statistical Estimation Under Distribution Shift: Wasserstein Perturbations and Minimax Theory [24.540342159350015]
  We focus on Wasserstein distribution shifts, where every data point may undergo a slight perturbation. We consider perturbations that are either independent or coordinated joint shifts across data points. We analyze several important statistical problems, including location estimation, linear regression, and non-parametric density estimation.
  arXiv  Detail & Related papers  (2023-08-03T16:19:40Z)
• Last-Iterate Convergence of Adaptive Riemannian Gradient Descent for Equilibrium Computation [52.73824786627612]
  This paper establishes new convergence results for textitgeodesic strongly monotone games.<n>Our key result shows that RGD attains last-iterate linear convergence in a textitgeometry-agnostic fashion.<n>Overall, this paper presents the first geometry-agnostic last-iterate convergence analysis for games beyond the Euclidean settings.
  arXiv  Detail & Related papers  (2023-06-29T01:20:44Z)
• The Inductive Bias of Flatness Regularization for Deep Matrix Factorization [58.851514333119255]
  This work takes the first step toward understanding the inductive bias of the minimum trace of the Hessian solutions in deep linear networks. We show that for all depth greater than one, with the standard Isometry Property (RIP) on the measurements, minimizing the trace of Hessian is approximately equivalent to minimizing the Schatten 1-norm of the corresponding end-to-end matrix parameters.
  arXiv  Detail & Related papers  (2023-06-22T23:14:57Z)
• On the Variance, Admissibility, and Stability of Empirical Risk Minimization [57.63331017830154]
  Empirical Risk Minimization (ERM) may attain minimax suboptimal rates in terms of the mean squared error.<n>We prove that under relatively mild assumptions, the suboptimality of ERM must be due to its large bias.
  arXiv  Detail & Related papers  (2023-05-29T15:25:48Z)
• The Geometry of Neural Nets' Parameter Spaces Under Reparametrization [35.5848464226014]
  We study the invariance of neural nets under reparametrization from the perspective of Riemannian geometry. We discuss implications for measuring the flatness of minima, optimization, and for probability-density.
  arXiv  Detail & Related papers  (2023-02-14T22:48:24Z)
• On the Double Descent of Random Features Models Trained with SGD [78.0918823643911]
  We study properties of random features (RF) regression in high dimensions optimized by gradient descent (SGD) We derive precise non-asymptotic error bounds of RF regression under both constant and adaptive step-size SGD setting. We observe the double descent phenomenon both theoretically and empirically.
  arXiv  Detail & Related papers  (2021-10-13T17:47:39Z)
• Optimal policy evaluation using kernel-based temporal difference methods [78.83926562536791]
  We use kernel Hilbert spaces for estimating the value function of an infinite-horizon discounted Markov reward process. We derive a non-asymptotic upper bound on the error with explicit dependence on the eigenvalues of the associated kernel operator. We prove minimax lower bounds over sub-classes of MRPs.
  arXiv  Detail & Related papers  (2021-09-24T14:48:20Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• Entropic gradient descent algorithms and wide flat minima [6.485776570966397]
  We show analytically that there exist Bayes optimal pointwise estimators which correspond to minimizers belonging to wide flat regions. We extend the analysis to the deep learning scenario by extensive numerical validations. An easy to compute flatness measure shows a clear correlation with test accuracy.
  arXiv  Detail & Related papers  (2020-06-14T13:22:19Z)
• The Heavy-Tail Phenomenon in SGD [7.366405857677226]
  We show that depending on the structure of the Hessian of the loss at the minimum, the SGD iterates will converge to a emphheavy-tailed stationary distribution. We translate our results into insights about the behavior of SGD in deep learning.
  arXiv  Detail & Related papers  (2020-06-08T16:43:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.