Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization

• URL: http://arxiv.org/abs/2607.18306v1
• Date: Fri, 17 Jul 2026 06:59:02 GMT
• Title: Gradient-Energy Guided Block-Wise Perturbations for Sharpness-Aware Minimization
• Abstract summary: GEAR-SAM maintains an exponential moving average (EMA) of squared block gradients as a lightweight, curvature-related sensitivity signal.<n>Experiments on image classification, transfer learning, noisy-label learning, and partition studies demonstrate improved generalization and robustness across architectures and tasks.
• Score: 11.183574804029165
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Sharpness-Aware Minimization (SAM) improves generalization by minimizing the worst-case loss in a local parameter neighborhood. Standard SAM implicitly allocates its global perturbation budget across parameter blocks according to instantaneous minibatch gradient norms. Such an allocation can be noisy and may not reflect the sensitivity that blocks accumulate throughout training. We propose Gradient-Energy Adaptive Radius SAM (GEAR-SAM), which maintains an exponential moving average (EMA) of squared block gradients as a lightweight, curvature-related sensitivity signal and allocates the fixed SAM budget through a closed-form constrained optimization. GEAR-SAM preserves the global SAM radius, requires no Hessian-vector products or explicit Fisher estimation, and adds only scalar state beyond SAM. Experiments on image classification, transfer learning, noisy-label learning, and partition studies demonstrate improved generalization and robustness across architectures and tasks. More broadly, GEAR-SAM provides a dynamic view of sharpness-aware optimization: a fixed perturbation budget should be redistributed as the sensitivity of functional network blocks evolves during training.

Related papers

• EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations [1.7073691799016324]
  Sharpness-Aware Minimization (SAM) was proposed as a training strategy that enhances generalization.<n>SAM suffers from its twice computational cost due to its core algorithm requiring an extra gradient during the perturbation step.<n>We introduce Exponential Moving Average Sharpness-Aware Minimization (EMASAM), a computationally efficient variant of SAM.
  arXiv  Detail & Related papers  (2026-08-15T08:10:31Z)
• Fix the Loss, Not the Radius: Rethinking the Adversarial Perturbation of Sharpness-Aware Minimization [7.865097630039961]
  Loss-Equated SAM (LE-SAM) is a generalization of Sharpness-Aware Minimization (SAM) that removes gradient-norm-dominated learning signals and shifts optimization toward curvature-dominated terms.<n>Le-SAM consistently outperforms SAM and even its variants, achieving the state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-05-11T08:34:36Z)
• ZO-SAM: Zero-Order Sharpness-Aware Minimization for Efficient Sparse Training [22.67717083974045]
  Sparse neural networks significantly alleviate constraints by dramatically reducing parameter count and computational overhead.<n>Existing sparse training methods often experience chaotic and noisy gradient signals, severely hindering convergence and generalization performance.<n>We propose Zero-Order Sharpness-Aware Minimization (ZO-SAM), a novel optimization framework that strategically integrates zero-order optimization within the SAM approach.
  arXiv  Detail & Related papers  (2026-03-13T16:11:41Z)
• Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning [52.63618112418439]
  Sharpness-aware computation (SAM) seeks the minima with a flat loss landscape to improve the generalization performance in machine learning tasks, including fine-tuning.<n>We propose an approach SL-SAM to break this bottleneck by introducing the sparse technique to layers.
  arXiv  Detail & Related papers  (2026-02-10T04:05:43Z)
• LightSAM: Parameter-Agnostic Sharpness-Aware Minimization [92.17866492331524]
  Sharpness-Aware Minimization (SAM) enhances the ability of the machine learning model by exploring the flat minima landscape through weight perturbations.<n>SAM introduces an additional hyper- parameter, the perturbation radius, which causes the sensitivity of SAM to it.<n>In this paper, we propose the algorithm LightSAM which sets the perturbation radius and learning rate of SAM adaptively.
  arXiv  Detail & Related papers  (2025-05-30T09:28:38Z)
• Friendly Sharpness-Aware Minimization [62.57515991835801]
  Sharpness-Aware Minimization (SAM) has been instrumental in improving deep neural network training by minimizing both training loss and loss sharpness. We investigate the key role of batch-specific gradient noise within the adversarial perturbation, i.e., the current minibatch gradient. By decomposing the adversarial gradient noise components, we discover that relying solely on the full gradient degrades generalization while excluding it leads to improved performance.
  arXiv  Detail & Related papers  (2024-03-19T01:39:33Z)
• Momentum-SAM: Sharpness Aware Minimization without Computational Overhead [1.5129569747339653]
  We propose Momentum-SAM, which perturbs parameters in the direction of the accumulated momentum vector to achieve low sharpness without significant computational overhead or memory demands.<n>We evaluate MSAM in detail and reveal insights on separable mechanisms of NAG, SAM and MSAM regarding training optimization and generalization.
  arXiv  Detail & Related papers  (2024-01-22T15:19:18Z)
• Enhancing Sharpness-Aware Optimization Through Variance Suppression [48.908966673827734]
  This work embraces the geometry of the loss function, where neighborhoods of 'flat minima' heighten generalization ability. It seeks 'flat valleys' by minimizing the maximum loss caused by an adversary perturbing parameters within the neighborhood. Although critical to account for sharpness of the loss function, such an 'over-friendly adversary' can curtail the outmost level of generalization.
  arXiv  Detail & Related papers  (2023-09-27T13:18:23Z)
• Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer [158.2634766682187]
  Deep neural networks often suffer from poor generalization due to complex and non- unstructured loss landscapes. SharpnessAware Minimization (SAM) is a popular solution that smooths the loss by minimizing the change of landscape when adding a perturbation. In this paper, we propose Sparse SAM (SSAM), an efficient and effective training scheme that achieves perturbation by a binary mask.
  arXiv  Detail & Related papers  (2023-06-30T09:33:41Z)
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks [146.2011175973769]
  This paper proposes Efficient Sharpness Aware Minimizer (M) which boosts SAM s efficiency at no cost to its generalization performance. M includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. We show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis bases.
  arXiv  Detail & Related papers  (2021-10-07T02:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.