ZO-SAM: Zero-Order Sharpness-Aware Minimization for Efficient Sparse Training

• URL: http://arxiv.org/abs/2603.13115v1
• Date: Fri, 13 Mar 2026 16:11:41 GMT
• Title: ZO-SAM: Zero-Order Sharpness-Aware Minimization for Efficient Sparse Training
• Abstract summary: Sparse neural networks significantly alleviate constraints by dramatically reducing parameter count and computational overhead.<n>Existing sparse training methods often experience chaotic and noisy gradient signals, severely hindering convergence and generalization performance.<n>We propose Zero-Order Sharpness-Aware Minimization (ZO-SAM), a novel optimization framework that strategically integrates zero-order optimization within the SAM approach.
• Score: 22.67717083974045
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning models, despite their impressive achievements, suffer from high computational costs and memory requirements, limiting their usability in resource-constrained environments. Sparse neural networks significantly alleviate these constraints by dramatically reducing parameter count and computational overhead. However, existing sparse training methods often experience chaotic and noisy gradient signals, severely hindering convergence and generalization performance, particularly at high sparsity levels. To tackle this critical challenge, we propose Zero-Order Sharpness-Aware Minimization (ZO-SAM), a novel optimization framework that strategically integrates zero-order optimization within the SAM approach. Unlike traditional SAM, ZO-SAM requires only a single backpropagation step during perturbation, selectively utilizing zero-order gradient estimations. This innovative approach reduces the backpropagation computational cost by half compared to conventional SAM, significantly lowering gradient variance and effectively eliminating associated computational overhead. By harnessing SAM's capacity for identifying flat minima, ZO-SAM stabilizes the training process and accelerates convergence. These efficiency gains are particularly important in sparse training scenarios, where computational cost is the primary bottleneck that limits the practicality of SAM. Moreover, models trained with ZO-SAM exhibit improved robustness under distribution shift, further broadening its practicality in real-world deployments.

Related papers

• EMASAM: a Computationally Efficient Sharpness-Aware Minimization via EMA-Guided Perturbations [1.7073691799016324]
  Sharpness-Aware Minimization (SAM) was proposed as a training strategy that enhances generalization.<n>SAM suffers from its twice computational cost due to its core algorithm requiring an extra gradient during the perturbation step.<n>We introduce Exponential Moving Average Sharpness-Aware Minimization (EMASAM), a computationally efficient variant of SAM.
  arXiv  Detail & Related papers  (2026-08-15T08:10:31Z)
• Sparse Layer Sharpness-Aware Minimization for Efficient Fine-Tuning [52.63618112418439]
  Sharpness-aware computation (SAM) seeks the minima with a flat loss landscape to improve the generalization performance in machine learning tasks, including fine-tuning.<n>We propose an approach SL-SAM to break this bottleneck by introducing the sparse technique to layers.
  arXiv  Detail & Related papers  (2026-02-10T04:05:43Z)
• LSAM: Asynchronous Distributed Training with Landscape-Smoothed Sharpness-Aware Minimization [6.794145254474338]
  Sharpness-Aware Minimization (SAM) improves generalization in deep neural networks by minimizing both loss and sharpness.<n>We present Landscape-Smoothed SAM (LSAM), a novel generalization that preserves SAM's advantages while offering superior efficiency.
  arXiv  Detail & Related papers  (2025-09-03T08:07:43Z)
• LORENZA: Enhancing Generalization in Low-Rank Gradient LLM Training via Efficient Zeroth-Order Adaptive SAM [13.180761892449736]
  We study robust parameter-efficient fine-tuning (PEFT) techniques for large-language models (LLMs)<n>We present a new highly computationally efficient framework called AdaZo-SAM, combining Adam and Sharpness-Aware Minimization (SAM)<n>We also design a low-rank gradient optimization method named LORENZA, which is a memory-efficient version of AdaZo-SAM.
  arXiv  Detail & Related papers  (2025-02-26T21:30:34Z)
• Stabilizing Sharpness-aware Minimization Through A Simple Renormalization Strategy [12.050160495730381]
  sharpness-aware generalization (SAM) has attracted much attention because of its surprising effectiveness in improving performance. We propose a simple renormalization strategy, dubbed Stable SAM (SSAM), so that the gradient norm of the descent step maintains the same as that of the ascent step. Our strategy is easy to implement and flexible enough to integrate with SAM and its variants, almost at no computational cost.
  arXiv  Detail & Related papers  (2024-01-14T10:53:36Z)
• Systematic Investigation of Sparse Perturbed Sharpness-Aware Minimization Optimizer [158.2634766682187]
  Deep neural networks often suffer from poor generalization due to complex and non- unstructured loss landscapes. SharpnessAware Minimization (SAM) is a popular solution that smooths the loss by minimizing the change of landscape when adding a perturbation. In this paper, we propose Sparse SAM (SSAM), an efficient and effective training scheme that achieves perturbation by a binary mask.
  arXiv  Detail & Related papers  (2023-06-30T09:33:41Z)
• K-SAM: Sharpness-Aware Minimization at the Speed of SGD [83.78737278889837]
  Sharpness-Aware Minimization (SAM) has emerged as a robust technique for improving the accuracy of deep neural networks. SAM incurs a high computational cost in practice, requiring up to twice as much computation as vanilla SGD. We propose to compute gradients in both stages of SAM on only the top-k samples with highest loss.
  arXiv  Detail & Related papers  (2022-10-23T21:49:58Z)
• Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach [132.37966970098645]
  One of the popular solutions is Sharpness-Aware Minimization (SAM), which minimizes the change of weight loss when adding a perturbation. In this paper, we propose an efficient effective training scheme coined as Sparse SAM (SSAM), which achieves double overhead of common perturbations. In addition, we theoretically prove that S can converge at the same SAM, i.e., $O(log T/sqrtTTTTTTTTTTTTTTTTT
  arXiv  Detail & Related papers  (2022-10-11T06:30:10Z)
• Sharpness-Aware Training for Free [163.1248341911413]
  SharpnessAware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. Sharpness-Aware Training Free (SAF) mitigates the sharp landscape at almost zero computational cost over the base. SAF ensures the convergence to a flat minimum with improved capabilities.
  arXiv  Detail & Related papers  (2022-05-27T16:32:43Z)
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks [146.2011175973769]
  This paper proposes Efficient Sharpness Aware Minimizer (M) which boosts SAM s efficiency at no cost to its generalization performance. M includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. We show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis bases.
  arXiv  Detail & Related papers  (2021-10-07T02:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.