Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning

• URL: http://arxiv.org/abs/2607.06151v1
• Date: Tue, 07 Jul 2026 11:25:46 GMT
• Title: Leveraging Extragradient for Effective Sharpness-Aware Minimization in Deep Learning
• Abstract summary: Generalization remains a pivotal challenge in deep learning, where traditional datasets converge to sharp minima, leading to overfitting and reduced performance on unseen data.<n>We propose the Extragradient-Inspired Sharpness-Aware Minimization (EISAM), a novel that enhances generalization via the extragradient technique.
• Score: 31.961129763228175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generalization remains a pivotal challenge in deep learning, where traditional optimizers like Stochastic Gradient Descent (SGD) often converge to sharp minima, leading to overfitting and reduced performance on unseen data. Building on Sharpness-Aware Minimization (SAM), for seeking flat minima associated with improved generalization, we propose the Extragradient-Inspired Sharpness-Aware Minimization (EISAM), a novel optimizer that enhances generalization via the extragradient technique. EISAM uses a two-step update process: a prediction step investigating the geometry of the loss landscape and a perturbation step that refines updates with a base optimizer. This approach achieves better generalization performance than SAM. Crucially, EISAM reduces sensitivity to the perturbation radius, enhancing robustness, and simplifying the tuning across diverse settings. Extensive experiments on benchmark datasets demonstrate that EISAM consistently outperforms SGD, Adaptive Moment Estimation (Adam), and SAM in test accuracy and training efficiency across various architectures. Theoretical analysis further confirms that EISAM tightens the generalization bound by steering parameters toward flatter minima with reduced curvature. Accompanied by a thorough hyperparameter analysis, EISAM offers practical tuning guidance, establishing it as a robust, scalable, and broadly applicable optimization solution that advances both the theory and practice in deep learning.

Related papers

• Adaptive Sharpness-Aware Minimization with a Polyak-type Step size: A Theory-Grounded Scheduler [16.39606116102731]
  Sharpness-Aware Minimization (SAM) has established itself as a powerful and widely adopted machine learning models.<n>SAM and its variants, like most training algorithms, are sensitive to the choice of learning rate.<n>In this work, we derive Polyak schedulers tailored to SAM-style updates.<n>Experiments demonstrate that the proposed Polyak schedulers achieve performance comparable to or better than carefully tuned SAM baselines.
  arXiv  Detail & Related papers  (2026-06-01T07:42:01Z)
• LightSAM: Parameter-Agnostic Sharpness-Aware Minimization [92.17866492331524]
  Sharpness-Aware Minimization (SAM) enhances the ability of the machine learning model by exploring the flat minima landscape through weight perturbations.<n>SAM introduces an additional hyper- parameter, the perturbation radius, which causes the sensitivity of SAM to it.<n>In this paper, we propose the algorithm LightSAM which sets the perturbation radius and learning rate of SAM adaptively.
  arXiv  Detail & Related papers  (2025-05-30T09:28:38Z)
• GCSAM: Gradient Centralized Sharpness Aware Minimization [45.05109291721135]
  Sharpness-Aware Minimization (SAM) has emerged as an effective optimization technique for reducing the sharpness of the loss landscape.<n>We propose Gradient-Sharpness-Aware Minimization (GCSAM), which incorporates Gradient Centralization (GC) to stabilize and accelerate convergence.<n>GCSAM consistently outperforms SAM and the Adam in terms of generalization and computational efficiency.
  arXiv  Detail & Related papers  (2025-01-20T16:42:31Z)
• 1st-Order Magic: Analysis of Sharpness-Aware Minimization [0.0]
  Sharpness-Aware Minimization (SAM) is an optimization technique designed to improve generalization by favoring flatter loss minima. We find that more precise approximations of the proposed SAM objective degrade generalization performance. This highlights a gap in our understanding of SAM's effectiveness and calls for further investigation into the role of approximations in optimization.
  arXiv  Detail & Related papers  (2024-11-03T23:50:34Z)
• Friendly Sharpness-Aware Minimization [62.57515991835801]
  Sharpness-Aware Minimization (SAM) has been instrumental in improving deep neural network training by minimizing both training loss and loss sharpness. We investigate the key role of batch-specific gradient noise within the adversarial perturbation, i.e., the current minibatch gradient. By decomposing the adversarial gradient noise components, we discover that relying solely on the full gradient degrades generalization while excluding it leads to improved performance.
  arXiv  Detail & Related papers  (2024-03-19T01:39:33Z)
• Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy [12.050160495730381]
  sharpness-aware generalization (SAM) has attracted much attention because of its surprising effectiveness in improving performance. We propose a simple renormalization strategy, dubbed Stable SAM (SSAM), so that the gradient norm of the descent step maintains the same as that of the ascent step. Our strategy is easy to implement and flexible enough to integrate with SAM and its variants, almost at no computational cost.
  arXiv  Detail & Related papers  (2024-01-14T10:53:36Z)
• Critical Influence of Overparameterization on Sharpness-aware Minimization [12.321517302762558]
  Sharpness-Aware Minimization (SAM) has attracted considerable attention for its effectiveness in improving generalization in deep neural network training.<n>This work presents both empirical and theoretical findings that reveal its critical influence on SAM's effectiveness.
  arXiv  Detail & Related papers  (2023-11-29T11:19:50Z)
• Normalization Layers Are All That Sharpness-Aware Minimization Needs [53.799769473526275]
  Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima. We show that perturbing only the affine normalization parameters (typically comprising 0.1% of the total parameters) in the adversarial step of SAM can outperform perturbing all of the parameters.
  arXiv  Detail & Related papers  (2023-06-07T08:05:46Z)
• AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks [76.90477930208982]
  Sharpness aware (SAM) has been extensively explored as it can generalize better for training deep neural networks. Integrating SAM with adaptive learning perturbation and momentum acceleration, dubbed AdaSAM, has already been explored. We conduct several experiments on several NLP tasks, which show that AdaSAM could achieve superior performance compared with SGD, AMS, and SAMsGrad.
  arXiv  Detail & Related papers  (2023-03-01T15:12:42Z)
• Sharpness-Aware Training for Free [163.1248341911413]
  SharpnessAware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. Sharpness-Aware Training Free (SAF) mitigates the sharp landscape at almost zero computational cost over the base. SAF ensures the convergence to a flat minimum with improved capabilities.
  arXiv  Detail & Related papers  (2022-05-27T16:32:43Z)
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks [146.2011175973769]
  This paper proposes Efficient Sharpness Aware Minimizer (M) which boosts SAM s efficiency at no cost to its generalization performance. M includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. We show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis bases.
  arXiv  Detail & Related papers  (2021-10-07T02:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.