Critical Influence of Overparameterization on Sharpness-aware Minimization

• URL: http://arxiv.org/abs/2311.17539v4
• Date: Sat, 15 Mar 2025 06:30:49 GMT
• Title: Critical Influence of Overparameterization on Sharpness-aware Minimization
• Authors: Sungbin Shin, Dongyeop Lee, Maksym Andriushchenko, Namhoon Lee,
• Abstract summary: We show that sharpness-aware minimization (SAM) is affected by over parameterization.<n>We show that this effect is particularly pronounced in practical settings involving label noise and sparsity.<n>We also provide insights into how over parameterization helps SAM achieve minima with more uniform Hessian moments compared to SGD.
• Score: 12.321517302762558
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training overparameterized neural networks often yields solutions with varying generalization capabilities, even when achieving similar training losses. Recent evidence indicates a strong correlation between the sharpness of a minimum and its generalization error, leading to increased interest in optimization methods that explicitly seek flatter minima for improved generalization. Despite its contemporary relevance to overparameterization, however, this sharpness-aware minimization (SAM) strategy has not been studied much yet as to exactly how it is affected by overparameterization. In this work, we analyze SAM under varying degrees of overparameterization, presenting both empirical and theoretical findings that reveal its critical influence on SAM's effectiveness. First, we conduct extensive numerical experiments across diverse domains, demonstrating that SAM consistently benefits from overparameterization. Next, we attribute this phenomenon to the interplay between the enlarged solution space and increased implicit bias resulting from overparameterization. Furthermore, we show that this effect is particularly pronounced in practical settings involving label noise and sparsity, and yet, sufficient regularization is necessary. Last but not least, we provide other theoretical insights into how overparameterization helps SAM achieve minima with more uniform Hessian moments compared to SGD, and much faster convergence at a linear rate.

Related papers

• Sharpness-Aware Minimization: General Analysis and Improved Rates [10.11126899274029]
  Sharpness-Aware Minimization (SAM) has emerged as a powerful method for improving generalization in machine learning models. We provide an analysis of SAM and its unnormalized variant rule rule (USAM) under one update. We present results of the new size under a relaxed more natural assumption.
  arXiv  Detail & Related papers  (2025-03-04T03:04:06Z)
• Monge SAM: Robust Reparameterization-Invariant Sharpness-Aware Minimization Based on Loss Geometry [2.854482269849925]
  Sharpness-aware minimization (SAM) efficiently finds flat regions by updating the parameters according to the gradient at an adversarial perturbation. We propose Monge SAM (M-SAM), a reparametrization invariant version of SAM. We demonstrate this behavior both theoretically and empirically on a multi-modal representation alignment task.
  arXiv  Detail & Related papers  (2025-02-12T14:40:19Z)
• $\boldsymbolμ\mathbf{P^2}$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling [49.25546155981064]
  We study the infinite-width limit of neural networks trained with Sharpness Aware Minimization (SAM) Our findings reveal that the dynamics of standard SAM effectively reduce to applying SAM solely in the last layer in wide neural networks. In contrast, we identify a stable parameterization with layerwise scaling perturbation, which we call $textitMaximal Update and Perturbation $ ($mu$P$2$), that ensures all layers are both feature learning and effectively perturbed in the limit.
  arXiv  Detail & Related papers  (2024-10-31T16:32:04Z)
• A Universal Class of Sharpness-Aware Minimization Algorithms [57.29207151446387]
  We introduce a new class of sharpness measures, leading to new sharpness-aware objective functions. We prove that these measures are textitly expressive, allowing any function of the training loss Hessian matrix to be represented by appropriate hyper and determinants.
  arXiv  Detail & Related papers  (2024-06-06T01:52:09Z)
• Normalization Layers Are All That Sharpness-Aware Minimization Needs [53.799769473526275]
  Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima. We show that perturbing only the affine normalization parameters (typically comprising 0.1% of the total parameters) in the adversarial step of SAM can outperform perturbing all of the parameters.
  arXiv  Detail & Related papers  (2023-06-07T08:05:46Z)
• The Crucial Role of Normalization in Sharpness-Aware Minimization [44.00155917998616]
  Sharpness-Aware Minimization (SAM) is a gradient-based neural network that greatly improves prediction performance. We argue that two properties of normalization make SAM robust against the choice of hyper- practicalitys.
  arXiv  Detail & Related papers  (2023-05-24T16:09:41Z)
• AdaSAM: Boosting Sharpness-Aware Minimization with Adaptive Learning Rate and Momentum for Training Deep Neural Networks [76.90477930208982]
  Sharpness aware (SAM) has been extensively explored as it can generalize better for training deep neural networks. Integrating SAM with adaptive learning perturbation and momentum acceleration, dubbed AdaSAM, has already been explored. We conduct several experiments on several NLP tasks, which show that AdaSAM could achieve superior performance compared with SGD, AMS, and SAMsGrad.
  arXiv  Detail & Related papers  (2023-03-01T15:12:42Z)
• Stability Analysis of Sharpness-Aware Minimization [5.024497308975435]
  Sharpness-aware (SAM) is a recently proposed training method that seeks to find flat minima in deep learning. In this paper, we demonstrate that SAM dynamics can have convergence instability that occurs near a saddle point.
  arXiv  Detail & Related papers  (2023-01-16T08:42:40Z)
• Sharpness-Aware Training for Free [163.1248341911413]
  SharpnessAware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. Sharpness-Aware Training Free (SAF) mitigates the sharp landscape at almost zero computational cost over the base. SAF ensures the convergence to a flat minimum with improved capabilities.
  arXiv  Detail & Related papers  (2022-05-27T16:32:43Z)
• Parameters or Privacy: A Provable Tradeoff Between Overparameterization and Membership Inference [29.743945643424553]
  Over parameterized models generalize well (small error on the test data) even when trained to memorize the training data (zero error on the training data) This has led to an arms race towards increasingly over parameterized models (c.f., deep learning)
  arXiv  Detail & Related papers  (2022-02-02T19:00:21Z)
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks [146.2011175973769]
  This paper proposes Efficient Sharpness Aware Minimizer (M) which boosts SAM s efficiency at no cost to its generalization performance. M includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. We show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis bases.
  arXiv  Detail & Related papers  (2021-10-07T02:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.