DGSAM: Domain Generalization via Individual Sharpness-Aware Minimization

• URL: http://arxiv.org/abs/2503.23430v1
• Date: Sun, 30 Mar 2025 13:20:06 GMT
• Title: DGSAM: Domain Generalization via Individual Sharpness-Aware Minimization
• Authors: Youngjun Song, Youngsik Hwang, Jonghun Lee, Heechang Lee, Dong-Young Lim,
• Abstract summary: Domain generalization (DG) aims to learn models that can generalize well to unseen domains by training only on a set of source domains.<n>Sharpness-Aware Minimization (SAM) has been a popular approach for this, aiming to find flat minima in the total loss landscape.<n>We introduce a novel DG algorithm, Decreased-overhead Gradual Sharpness-Aware Minimization (DGSAM), that applies gradual domain-wise perturbation to reduce sharpness consistently across domains.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Domain generalization (DG) aims to learn models that can generalize well to unseen domains by training only on a set of source domains. Sharpness-Aware Minimization (SAM) has been a popular approach for this, aiming to find flat minima in the total loss landscape. However, we show that minimizing the total loss sharpness does not guarantee sharpness across individual domains. In particular, SAM can converge to fake flat minima, where the total loss may exhibit flat minima, but sharp minima are present in individual domains. Moreover, the current perturbation update in gradient ascent steps is ineffective in directly updating the sharpness of individual domains. Motivated by these findings, we introduce a novel DG algorithm, Decreased-overhead Gradual Sharpness-Aware Minimization (DGSAM), that applies gradual domain-wise perturbation to reduce sharpness consistently across domains while maintaining computational efficiency. Our experiments demonstrate that DGSAM outperforms state-of-the-art DG methods, achieving improved robustness to domain shifts and better performance across various benchmarks, while reducing computational overhead compared to SAM.

Related papers

• GCSAM: Gradient Centralized Sharpness Aware Minimization [45.05109291721135]
  Sharpness-Aware Minimization (SAM) has emerged as an effective optimization technique for reducing the sharpness of the loss landscape.<n>We propose Gradient-Sharpness-Aware Minimization (GCSAM), which incorporates Gradient Centralization (GC) to stabilize and accelerate convergence.<n>GCSAM consistently outperforms SAM and the Adam in terms of generalization and computational efficiency.
  arXiv  Detail & Related papers  (2025-01-20T16:42:31Z)
• Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts [38.78485556098491]
  This paper presents a Domain-Inspired Sharpness-Aware Minimization (DISAM) algorithm for optimization under domain shifts. It is motivated by the inconsistent convergence degree of SAM across different domains, which induces optimization bias towards certain domains. Under this mechanism, we theoretically show that DISAM can achieve faster overall convergence and improved generalization in principle.
  arXiv  Detail & Related papers  (2024-05-29T08:22:33Z)
• Friendly Sharpness-Aware Minimization [62.57515991835801]
  Sharpness-Aware Minimization (SAM) has been instrumental in improving deep neural network training by minimizing both training loss and loss sharpness. We investigate the key role of batch-specific gradient noise within the adversarial perturbation, i.e., the current minibatch gradient. By decomposing the adversarial gradient noise components, we discover that relying solely on the full gradient degrades generalization while excluding it leads to improved performance.
  arXiv  Detail & Related papers  (2024-03-19T01:39:33Z)
• Normalization Layers Are All That Sharpness-Aware Minimization Needs [53.799769473526275]
  Sharpness-aware minimization (SAM) was proposed to reduce sharpness of minima. We show that perturbing only the affine normalization parameters (typically comprising 0.1% of the total parameters) in the adversarial step of SAM can outperform perturbing all of the parameters.
  arXiv  Detail & Related papers  (2023-06-07T08:05:46Z)
• Sharpness-Aware Gradient Matching for Domain Generalization [84.14789746460197]
  The goal of domain generalization (DG) is to enhance the generalization capability of the model learned from a source domain to other unseen domains. The recently developed Sharpness-Aware Minimization (SAM) method aims to achieve this goal by minimizing the sharpness measure of the loss landscape. We present two conditions to ensure that the model could converge to a flat minimum with a small loss, and present an algorithm, named Sharpness-Aware Gradient Matching (SAGM) Our proposed SAGM method consistently outperforms the state-of-the-art methods on five DG benchmarks.
  arXiv  Detail & Related papers  (2023-03-18T07:25:12Z)
• Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalization [33.50116027503244]
  We show that the zeroth-order flatness can be insufficient to discriminate minima with low gradient error. We also present a novel training procedure named Gradient norm Aware Minimization (GAM) to seek minima with uniformly small curvature across all directions.
  arXiv  Detail & Related papers  (2023-03-03T16:58:53Z)
• Sharpness-Aware Training for Free [163.1248341911413]
  SharpnessAware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. Sharpness-Aware Training Free (SAF) mitigates the sharp landscape at almost zero computational cost over the base. SAF ensures the convergence to a flat minimum with improved capabilities.
  arXiv  Detail & Related papers  (2022-05-27T16:32:43Z)
• Surrogate Gap Minimization Improves Sharpness-Aware Training [52.58252223573646]
  Surrogate textbfGap Guided textbfSharpness-textbfAware textbfMinimization (GSAM) is a novel improvement over Sharpness-Aware Minimization (SAM) with negligible computation overhead. GSAM seeks a region with both small loss (by step 1) and low sharpness (by step 2), giving rise to a model with high generalization capabilities.
  arXiv  Detail & Related papers  (2022-03-15T16:57:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.