Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification

• URL: http://arxiv.org/abs/2505.01660v2
• Date: Tue, 03 Jun 2025 11:43:05 GMT
• Title: Focal-SAM: Focal Sharpness-Aware Minimization for Long-Tailed Classification
• Authors: Sicong Li, Qianqian Xu, Zhiyong Yang, Zitai Wang, Linchao Zhang, Xiaochun Cao, Qingming Huang,
• Abstract summary: Real-world datasets often follow a long-tailed distribution, making generalization to tail classes difficult.<n>Recent methods resorted to long-tail variants of Sharpness-Aware Minimization (SAM) to improve generalization by flattening the loss landscape.<n>We introduce Focal-SAM, which assigns different penalties to class-wise, achieving fine-grained control without extra backpropagations.
• Score: 113.6840565194525
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Real-world datasets often follow a long-tailed distribution, making generalization to tail classes difficult. Recent methods resorted to long-tail variants of Sharpness-Aware Minimization (SAM), such as ImbSAM and CC-SAM, to improve generalization by flattening the loss landscape. However, these attempts face a trade-off between computational efficiency and control over the loss landscape. On the one hand, ImbSAM is efficient but offers only coarse control as it excludes head classes from the SAM process. On the other hand, CC-SAM provides fine-grained control through class-dependent perturbations but at the cost of efficiency due to multiple backpropagations. Seeing this dilemma, we introduce Focal-SAM, which assigns different penalties to class-wise sharpness, achieving fine-grained control without extra backpropagations, thus maintaining efficiency. Furthermore, we theoretically analyze Focal-SAM's generalization ability and derive a sharper generalization bound. Extensive experiments on both traditional and foundation models validate the effectiveness of Focal-SAM.

Related papers

• SSE-SAM: Balancing Head and Tail Classes Gradually through Stage-Wise SAM [82.27086968525381]
  We argue that improving generalization in long-tail scenarios requires a careful balance between head and tail classes.<n>We show that neither SAM nor ImbSAM alone can fully achieve this balance.<n>We propose Stage-wise Saddle Escaping SAM (SSE-SAM), which uses complementary strengths of ImbSAM and SAM in a phased approach.
  arXiv  Detail & Related papers  (2024-12-18T10:53:36Z)
• Promptable Anomaly Segmentation with SAM Through Self-Perception Tuning [63.55145330447408]
  We propose a novel textbfSelf-textbfPerceptinon textbfTuning (textbfSPT) method for anomaly segmentation.<n>The SPT method incorporates a self-drafting tuning strategy, which generates an initial coarse draft of the anomaly mask, followed by a refinement process.
  arXiv  Detail & Related papers  (2024-11-26T08:33:25Z)
• Friendly Sharpness-Aware Minimization [62.57515991835801]
  Sharpness-Aware Minimization (SAM) has been instrumental in improving deep neural network training by minimizing both training loss and loss sharpness. We investigate the key role of batch-specific gradient noise within the adversarial perturbation, i.e., the current minibatch gradient. By decomposing the adversarial gradient noise components, we discover that relying solely on the full gradient degrades generalization while excluding it leads to improved performance.
  arXiv  Detail & Related papers  (2024-03-19T01:39:33Z)
• mSAM: Micro-Batch-Averaged Sharpness-Aware Minimization [20.560184120992094]
  Sharpness-Aware Minimization technique modifies the fundamental loss function that steers gradient descent methods toward flatter minima. We extend a recently developed and well-studied general framework for flatness analysis to theoretically show that SAM achieves flatter minima than SGD, and mSAM achieves even flatter minima than SAM.
  arXiv  Detail & Related papers  (2023-02-19T23:27:12Z)
• Improved Deep Neural Network Generalization Using m-Sharpness-Aware Minimization [14.40189851070842]
  Sharpness-Aware Minimization (SAM) modifies the underlying loss function to guide descent methods towards flatter minima. Recent work suggests that mSAM can outperform SAM in terms of test accuracy. This paper presents a comprehensive empirical evaluation of mSAM on various tasks and datasets.
  arXiv  Detail & Related papers  (2022-12-07T00:37:55Z)
• How Does Sharpness-Aware Minimization Minimize Sharpness? [29.90109733192208]
  Sharpness-Aware Minimization (SAM) is a highly effective regularization technique for improving the generalization of deep neural networks. This paper rigorously nails down the exact sharpness notion that SAM regularizes and clarifies the underlying mechanism.
  arXiv  Detail & Related papers  (2022-11-10T17:56:38Z)
• Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach [132.37966970098645]
  One of the popular solutions is Sharpness-Aware Minimization (SAM), which minimizes the change of weight loss when adding a perturbation. In this paper, we propose an efficient effective training scheme coined as Sparse SAM (SSAM), which achieves double overhead of common perturbations. In addition, we theoretically prove that S can converge at the same SAM, i.e., $O(log T/sqrtTTTTTTTTTTTTTTTTT
  arXiv  Detail & Related papers  (2022-10-11T06:30:10Z)
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks [146.2011175973769]
  This paper proposes Efficient Sharpness Aware Minimizer (M) which boosts SAM s efficiency at no cost to its generalization performance. M includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. We show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis bases.
  arXiv  Detail & Related papers  (2021-10-07T02:20:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.