Do Sharpness-based Optimizers Improve Generalization in Medical Image Analysis?

• URL: http://arxiv.org/abs/2408.04065v2
• Date: Fri, 9 Aug 2024 18:31:03 GMT
• Title: Do Sharpness-based Optimizers Improve Generalization in Medical Image Analysis?
• Authors: Mohamed Hassan, Aleksandar Vakanski, Min Xian,
• Abstract summary: Sharpness-Aware Minimization (SAM) has shown potential in enhancing generalization performance on general domain image datasets. This work provides a recent sharpness-based methods for improving the generalization of deep learning networks and evaluates the methods on medical breast ultrasound images.
• Score: 47.346907372319706
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Effective clinical deployment of deep learning models in healthcare demands high generalization performance to ensure accurate diagnosis and treatment planning. In recent years, significant research has focused on improving the generalization of deep learning models by regularizing the sharpness of the loss landscape. Among the optimization approaches that explicitly minimize sharpness, Sharpness-Aware Minimization (SAM) has shown potential in enhancing generalization performance on general domain image datasets. This success has led to the development of several advanced sharpness-based algorithms aimed at addressing the limitations of SAM, such as Adaptive SAM, surrogate-Gap SAM, Weighted SAM, and Curvature Regularized SAM. These sharpness-based optimizers have shown improvements in model generalization compared to conventional stochastic gradient descent optimizers and their variants on general domain image datasets, but they have not been thoroughly evaluated on medical images. This work provides a review of recent sharpness-based methods for improving the generalization of deep learning networks and evaluates the methods performance on medical breast ultrasound images. Our findings indicate that the initial SAM method successfully enhances the generalization of various deep learning models. While Adaptive SAM improves generalization of convolutional neural networks, it fails to do so for vision transformers. Other sharpness-based optimizers, however, do not demonstrate consistent results. The results reveal that, contrary to findings in the non-medical domain, SAM is the only recommended sharpness-based optimizer that consistently improves generalization in medical image analysis, and further research is necessary to refine the variants of SAM to enhance generalization performance in this field

Related papers

• GCSAM: Gradient Centralized Sharpness Aware Minimization [45.05109291721135]
  Sharpness-Aware Minimization (SAM) has emerged as an effective optimization technique for reducing the sharpness of the loss landscape. We propose Gradient-Sharpness-Aware Minimization (GCSAM), which incorporates Gradient Centralization (GC) to stabilize and accelerate convergence. GCSAM consistently outperforms SAM and the Adam in terms of generalization and computational efficiency.
  arXiv  Detail & Related papers  (2025-01-20T16:42:31Z)
• 1st-Order Magic: Analysis of Sharpness-Aware Minimization [0.0]
  Sharpness-Aware Minimization (SAM) is an optimization technique designed to improve generalization by favoring flatter loss minima. We find that more precise approximations of the proposed SAM objective degrade generalization performance. This highlights a gap in our understanding of SAM's effectiveness and calls for further investigation into the role of approximations in optimization.
  arXiv  Detail & Related papers  (2024-11-03T23:50:34Z)
• Optimizing Universal Lesion Segmentation: State Space Model-Guided Hierarchical Networks with Feature Importance Adjustment [0.0]
  We introduce Mamba-Ahnet, a novel integration of State Space Model (SSM) and Advanced Hierarchical Network (AHNet) within the MAMBA framework. Mamba-Ahnet combines SSM's feature extraction and comprehension with AHNet's attention mechanisms and image reconstruction, aiming to enhance segmentation accuracy and robustness.
  arXiv  Detail & Related papers  (2024-04-26T08:15:43Z)
• Friendly Sharpness-Aware Minimization [62.57515991835801]
  Sharpness-Aware Minimization (SAM) has been instrumental in improving deep neural network training by minimizing both training loss and loss sharpness. We investigate the key role of batch-specific gradient noise within the adversarial perturbation, i.e., the current minibatch gradient. By decomposing the adversarial gradient noise components, we discover that relying solely on the full gradient degrades generalization while excluding it leads to improved performance.
  arXiv  Detail & Related papers  (2024-03-19T01:39:33Z)
• Model Generalization: A Sharpness Aware Optimization Perspective [4.017760528208121]
  Sharpness-Aware Minimization (SAM) and adaptive sharpness-aware minimization (ASAM) aim to improve the model generalization. Our experiments show that sharpness aware-based optimization techniques could help to provide models with strong generalization ability.
  arXiv  Detail & Related papers  (2022-08-14T20:50:17Z)
• Efficient Sharpness-aware Minimization for Improved Training of Neural Networks [146.2011175973769]
  This paper proposes Efficient Sharpness Aware Minimizer (M) which boosts SAM s efficiency at no cost to its generalization performance. M includes two novel and efficient training strategies-StochasticWeight Perturbation and Sharpness-Sensitive Data Selection. We show, via extensive experiments on the CIFAR and ImageNet datasets, that ESAM enhances the efficiency over SAM from requiring 100% extra computations to 40% vis-a-vis bases.
  arXiv  Detail & Related papers  (2021-10-07T02:20:37Z)
• On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy [70.71457102672545]
  We compare the impact of different training procedures for diabetic retinopathy grading. We investigate different aspects such as quantitative performance, statistics of the learned feature representations, interpretability and robustness to image distortions. Our results indicate that models from ImageNet pretraining report a significant increase in performance, generalization and robustness to image distortions.
  arXiv  Detail & Related papers  (2021-06-25T08:32:45Z)
• Evaluation of Complexity Measures for Deep Learning Generalization in Medical Image Analysis [77.34726150561087]
  PAC-Bayes flatness-based and path norm-based measures produce the most consistent explanation for the combination of models and data. We also investigate the use of multi-task classification and segmentation approach for breast images.
  arXiv  Detail & Related papers  (2021-03-04T20:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.