On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds

• URL: http://arxiv.org/abs/2608.03197v1
• Date: Tue, 04 Aug 2026 06:37:44 GMT
• Title: On the Implicit Flatness Bias of Sharpness-Aware Minimization: A Linear Stability Analysis with Quantitative Hyperparameter Bounds
• Abstract summary: Sharpness-Aware Minimization (SAM) improves generalization by seeking parameters whose loss is robust to local adversarial perturbations.<n>In particular, the perturbation radius $$ is typically treated as an isolated tuning parameter.<n>We analyze mini-batch SAM near an interpolating minimum through linear stability.
• Score: 7.33659299625996
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sharpness-Aware Minimization (SAM) improves generalization by seeking parameters whose loss is robust to local adversarial perturbations, but the quantitative mechanism underlying its implicit bias toward flat minima remains unclear. In particular, the perturbation radius $ρ$ is typically treated as an isolated tuning parameter, despite defining the neighborhood in which SAM measures sharpness. We analyze mini-batch SAM near an interpolating minimum through linear stability. Under local linearization and gradient-noise alignment assumptions, we prove that every linearly stable minimum satisfies $λ_{\max}\leq\sqrt[3]{bΓ/(2ρη^2)}$, where $λ_{\max}$ is the largest Hessian eigenvalue, $b$ is the batch size, $η$ is the learning rate, and $Γ$ bounds the gradient norm. The bound quantitatively characterizes SAM's implicit flatness bias: holding the other quantities fixed, a smaller batch size, a larger learning rate, or a larger radius restricts linearly stable SAM to flatter minima. It also exposes a necessary trade-off: $ρ$ should be large enough to promote flatness, yet remain local enough to preserve the approximation and stable training. We validate this prediction in a controlled study of 900 models on CIFAR-100 with ResNet-18 and VGG-19, where increasing $ρ$ is consistently associated with a smaller largest Hessian eigenvalue across batch-size and learning-rate settings. Finally, we instantiate the analysis in Taylor-Locality Controlled SAM (TLC-SAM), which adjusts $ρ$ using the observed Taylor-approximation error and further reduces the top Hessian eigenvalue relative to fixed-radius SAM. Our results provide quantitative hyperparameter bounds and a stability--locality perspective for analyzing and designing SAM variants.

Related papers

• Robust Average-Reward Markov Decision Processes: Minimax-Optimal Learning via Plug-in Reductions [51.50375419691955]
  Distributionally robust Markov decision processes provide a principled framework for sequential decision making under model uncertainty.<n>We study how many samples are necessary and sufficient to learn an $varepsilon$-optimal robust policy under the average-reward criterion.
  arXiv  Detail & Related papers  (2026-08-06T19:49:48Z)
• Zeroth-Order Optimization at the Edge of Stability [59.54782674222443]
  We provide an explicit step size condition that captures the (mean-square) linear stability of a family of Zeroth-order (ZO) methods.<n>Our results highlight an implicit regularization effect specific to ZO methods, where large step sizes primarily regularize the Hessian trace.
  arXiv  Detail & Related papers  (2026-04-16T06:23:18Z)
• X-SAM: Boosting Sharpness-Aware Minimization with Dominant-Eigenvector Gradient Correction [7.8091155908891965]
  We investigate Sharpness-Aware Minimization (SAM) from a spectral and geometric perspective.<n>We propose an explicit eigenvector-aligned SAM (X-SAM) which corrects the gradient via decomposition along the top eigenvector.<n>We prove X-SAM's convergence and superior generalization, with extensive experimental evaluations confirming both theoretical and practical advantages.
  arXiv  Detail & Related papers  (2026-01-15T10:19:08Z)
• Tilted Sharpness-Aware Minimization [24.689230137012174]
  Sharpness-Aware Minimization (SAM) has been demonstrated to improve the generalization performance of over infinity Hamiltonian generalization algorithms.<n>We show that TSAM is smoother than and thus easier to optimize than it is to solve TSAM.
  arXiv  Detail & Related papers  (2024-10-30T02:49:48Z)
• Bilateral Sharpness-Aware Minimization for Flatter Minima [61.17349662062522]
  Sharpness-Aware Minimization (SAM) enhances generalization by reducing a Max-Sharpness (MaxS) In this paper, we propose to utilize the difference between the training loss and the minimum loss over the neighborhood surrounding the current weight, which we denote as Min-Sharpness (MinS) By merging MaxS and MinS, we created a better FI that indicates a flatter direction during the optimization. Specially, we combine this FI with SAM into the proposed Bilateral SAM (BSAM) which finds a more flatter minimum than that of SAM.
  arXiv  Detail & Related papers  (2024-09-20T03:01:13Z)
• Practical Sharpness-Aware Minimization Cannot Converge All the Way to Optima [14.141453107129403]
  Sharpness-Aware Minimization (SAM) is an that takes a step based on a perturbation at a $y_t = x_t + rho fracbla f(x_t)lt blablax_t)
  arXiv  Detail & Related papers  (2023-06-16T13:47:04Z)
• Exact Mean Square Linear Stability Analysis for SGD [28.65663421598186]
  We provide an explicit condition on the step size that is both necessary and sufficient for linear stability of gradient descent (SGD) We show that SGD's stability threshold is equivalent to that of a mixture process which takes in each a full batch gradient step w.p. $1-p$, and a single sample gradient step w.p. $p$.
  arXiv  Detail & Related papers  (2023-06-13T15:29:23Z)
• Fisher SAM: Information Geometry and Sharpness Aware Minimisation [81.72160507751028]
  Recent sharpness-aware minimisation (SAM) is known to find flat minima. SAM essentially modifies the loss function by reporting the maximum loss value within the small neighborhood around the current iterate. We consider the information geometry of the model parameter space when defining the neighborhood, namely replacing SAM's Euclidean balls with ellipsoids induced by the Fisher information.
  arXiv  Detail & Related papers  (2022-06-10T07:42:51Z)
• Surrogate Gap Minimization Improves Sharpness-Aware Training [52.58252223573646]
  Surrogate textbfGap Guided textbfSharpness-textbfAware textbfMinimization (GSAM) is a novel improvement over Sharpness-Aware Minimization (SAM) with negligible computation overhead. GSAM seeks a region with both small loss (by step 1) and low sharpness (by step 2), giving rise to a model with high generalization capabilities.
  arXiv  Detail & Related papers  (2022-03-15T16:57:59Z)
• Robust Online Control with Model Misspecification [96.23493624553998]
  We study online control of an unknown nonlinear dynamical system with model misspecification. Our study focuses on robustness, which measures how much deviation from the assumed linear approximation can be tolerated.
  arXiv  Detail & Related papers  (2021-07-16T07:04:35Z)
• Finite-time Identification of Stable Linear Systems: Optimality of the Least-Squares Estimator [79.3239137440876]
  We present a new finite-time analysis of the estimation error of the Ordinary Least Squares (OLS) estimator for stable linear time-invariant systems. We characterize the number of observed samples sufficient for the OLS estimator to be $(varepsilon,delta)$-PAC, i.e., to yield an estimation error less than $varepsilon$ with probability at least $1-delta$.
  arXiv  Detail & Related papers  (2020-03-17T20:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.