Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition

• URL: http://arxiv.org/abs/2410.21042v1
• Date: Mon, 28 Oct 2024 13:58:17 GMT
• Title: Improving Visual Prompt Tuning by Gaussian Neighborhood Minimization for Long-Tailed Visual Recognition
• Authors: Mengke Li, Ye Liu, Yang Lu, Yiqun Zhang, Yiu-ming Cheung, Hui Huang,
• Abstract summary: We propose a novel method called Random SAM prompt tuning (RSAM-PT) to improve the model generalization. We employ the deferred re-weight scheme to increase the significance of tail-class samples. RSAM-PT achieves the state-of-the-art performance of 90.3%, 76.5%, and 50.1% on benchmark datasets.
• Score: 39.057791275055074
• License:
• Abstract: Long-tail learning has garnered widespread attention and achieved significant progress in recent times. However, even with pre-trained prior knowledge, models still exhibit weaker generalization performance on tail classes. The promising Sharpness-Aware Minimization (SAM) can effectively improve the generalization capability of models by seeking out flat minima in the loss landscape, which, however, comes at the cost of doubling the computational time. Since the update rule of SAM necessitates two consecutive (non-parallelizable) forward and backpropagation at each step. To address this issue, we propose a novel method called Random SAM prompt tuning (RSAM-PT) to improve the model generalization, requiring only one-step gradient computation at each step. Specifically, we search for the gradient descent direction within a random neighborhood of the parameters during each gradient update. To amplify the impact of tail-class samples and avoid overfitting, we employ the deferred re-weight scheme to increase the significance of tail-class samples. The classification accuracy of long-tailed data can be significantly improved by the proposed RSAM-PT, particularly for tail classes. RSAM-PT achieves the state-of-the-art performance of 90.3\%, 76.5\%, and 50.1\% on benchmark datasets CIFAR100-LT (IF 100), iNaturalist 2018, and Places-LT, respectively. The source code is temporarily available at https://github.com/Keke921/GNM-PT.

Related papers

• Efficient Sharpness-Aware Minimization for Molecular Graph Transformer Models [42.59948316941217]
  Sharpness-aware minimization (SAM) has received increasing attention in computer vision since it can effectively eliminate the sharp local minima from the training trajectory and generalization degradation. We propose a new algorithm named GraphSAM, which reduces the training cost of SAM and improves the generalization performance of graph transformer models.
  arXiv  Detail & Related papers  (2024-06-19T01:03:23Z)
• CR-SAM: Curvature Regularized Sharpness-Aware Minimization [8.248964912483912]
  Sharpness-Aware Minimization (SAM) aims to enhance the generalizability by minimizing worst-case loss using one-step gradient ascent as an approximation. In this paper, we introduce a normalized Hessian trace to accurately measure the curvature of loss landscape on em both training and test sets. In particular, to counter excessive non-linearity of loss landscape, we propose Curvature Regularized SAM (CR-SAM)
  arXiv  Detail & Related papers  (2023-12-21T03:46:29Z)
• ImbSAM: A Closer Look at Sharpness-Aware Minimization in Class-Imbalanced Recognition [62.20538402226608]
  We show that the Sharpness-Aware Minimization (SAM) fails to address generalization issues under the class-imbalanced setting. We propose a class-aware smoothness optimization algorithm named Imbalanced-SAM (ImbSAM) to overcome this bottleneck. Our ImbSAM demonstrates remarkable performance improvements for tail classes and anomaly.
  arXiv  Detail & Related papers  (2023-08-15T14:46:32Z)
• Consensus-Adaptive RANSAC [104.87576373187426]
  We propose a new RANSAC framework that learns to explore the parameter space by considering the residuals seen so far via a novel attention layer. The attention mechanism operates on a batch of point-to-model residuals, and updates a per-point estimation state to take into account the consensus found through a lightweight one-step transformer.
  arXiv  Detail & Related papers  (2023-07-26T08:25:46Z)
• Efficient Generalization Improvement Guided by Random Weight Perturbation [24.027159739234524]
  Gruesome-aware minimization (SAM) establishes a generic scheme for generalization improvements. We resort to filter-wise random weight perturbations (RWP) to decouple the nested gradients in SAM. We achieve very competitive performance on CIFAR and remarkably better performance on ImageNet.
  arXiv  Detail & Related papers  (2022-11-21T14:24:34Z)
• Sharpness-Aware Training for Free [163.1248341911413]
  SharpnessAware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. Sharpness-Aware Training Free (SAF) mitigates the sharp landscape at almost zero computational cost over the base. SAF ensures the convergence to a flat minimum with improved capabilities.
  arXiv  Detail & Related papers  (2022-05-27T16:32:43Z)
• Towards Efficient and Scalable Sharpness-Aware Minimization [81.22779501753695]
  We propose a novel algorithm LookSAM that only periodically calculates the inner gradient ascent. LookSAM achieves similar accuracy gains to SAM while being tremendously faster. We are the first to successfully scale up the batch size when training Vision Transformers (ViTs)
  arXiv  Detail & Related papers  (2022-03-05T11:53:37Z)
• Sharpness-Aware Minimization for Efficiently Improving Generalization [36.87818971067698]
  We introduce a novel, effective procedure for simultaneously minimizing loss value and loss sharpness. Sharpness-Aware Minimization (SAM) seeks parameters that lie in neighborhoods having uniformly low loss. We present empirical results showing that SAM improves model generalization across a variety of benchmark datasets.
  arXiv  Detail & Related papers  (2020-10-03T19:02:10Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.