Generalized Parametric Contrastive Learning

• URL: http://arxiv.org/abs/2209.12400v2
• Date: Sat, 27 May 2023 06:40:55 GMT
• Title: Generalized Parametric Contrastive Learning
• Authors: Jiequan Cui, Zhisheng Zhong, Zhuotao Tian, Shu Liu, Bei Yu, Jiaya Jia
• Abstract summary: Generalized Parametric Contrastive Learning (GPaCo/PaCo) works well on both imbalanced and balanced data. Experiments on long-tailed benchmarks manifest the new state-of-the-art for long-tailed recognition.
• Score: 60.62901294843829
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In this paper, we propose the Generalized Parametric Contrastive Learning (GPaCo/PaCo) which works well on both imbalanced and balanced data. Based on theoretical analysis, we observe that supervised contrastive loss tends to bias high-frequency classes and thus increases the difficulty of imbalanced learning. We introduce a set of parametric class-wise learnable centers to rebalance from an optimization perspective. Further, we analyze our GPaCo/PaCo loss under a balanced setting. Our analysis demonstrates that GPaCo/PaCo can adaptively enhance the intensity of pushing samples of the same class close as more samples are pulled together with their corresponding centers and benefit hard example learning. Experiments on long-tailed benchmarks manifest the new state-of-the-art for long-tailed recognition. On full ImageNet, models from CNNs to vision transformers trained with GPaCo loss show better generalization performance and stronger robustness compared with MAE models. Moreover, GPaCo can be applied to the semantic segmentation task and obvious improvements are observed on the 4 most popular benchmarks. Our code is available at https://github.com/dvlab-research/Parametric-Contrastive-Learning.

Related papers

• PACE: marrying generalization in PArameter-efficient fine-tuning with Consistency rEgularization [35.922096876707975]
  PACE is a generalization of PArameter-efficient fine-tuning with Consistency rEgularization. We show that PACE implicitly regularizes gradients for enhanced generalization, but also implicitly aligns the fine-tuned and pre-trained models to retain knowledge. PACE outperforms existing PEFT methods in four visual adaptation tasks: VTAB-1k, FGVC, few-shot learning and domain adaptation.
  arXiv  Detail & Related papers  (2024-09-25T17:56:00Z)
• Asymmetric Momentum: A Rethinking of Gradient Descent [4.1001738811512345]
  We propose the simplest SGD enhanced method, Loss-Controlled Asymmetric Momentum(LCAM) By averaging the loss, we divide training process into different loss phases and using different momentum. We experimentally validate that weights have directional specificity, which are correlated with the specificity of the dataset.
  arXiv  Detail & Related papers  (2023-09-05T11:16:47Z)
• Leveraging Angular Information Between Feature and Classifier for Long-tailed Learning: A Prediction Reformulation Approach [90.77858044524544]
  We reformulate the recognition probabilities through included angles without re-balancing the classifier weights. Inspired by the performance improvement of the predictive form reformulation, we explore the different properties of this angular prediction. Our method is able to obtain the best performance among peer methods without pretraining on CIFAR10/100-LT and ImageNet-LT.
  arXiv  Detail & Related papers  (2022-12-03T07:52:48Z)
• Improving Predictive Performance and Calibration by Weight Fusion in Semantic Segmentation [18.47581580698701]
  Averaging predictions of a deep ensemble of networks is a popular and effective method to improve predictive performance andcalibration. We show that a simple weight fusion (WF)strategy can lead to a significantly improved predictive performance andcalibration.
  arXiv  Detail & Related papers  (2022-07-22T17:24:13Z)
• Efficient Model-based Multi-agent Reinforcement Learning via Optimistic Equilibrium Computation [93.52573037053449]
  H-MARL (Hallucinated Multi-Agent Reinforcement Learning) learns successful equilibrium policies after a few interactions with the environment. We demonstrate our approach experimentally on an autonomous driving simulation benchmark.
  arXiv  Detail & Related papers  (2022-03-14T17:24:03Z)
• Relieving Long-tailed Instance Segmentation via Pairwise Class Balance [85.53585498649252]
  Long-tailed instance segmentation is a challenging task due to the extreme imbalance of training samples among classes. It causes severe biases of the head classes (with majority samples) against the tailed ones. We propose a novel Pairwise Class Balance (PCB) method, built upon a confusion matrix which is updated during training to accumulate the ongoing prediction preferences.
  arXiv  Detail & Related papers  (2022-01-08T07:48:36Z)
• Parametric Contrastive Learning [65.70554597097248]
  We propose Parametric Contrastive Learning (PaCo) to tackle long-tailed recognition. PaCo can adaptively enhance the intensity of pushing samples of the same class close. Experiments on long-tailed CIFAR, ImageNet, Places, and iNaturalist 2018 manifest the new state-of-the-art for long-tailed recognition.
  arXiv  Detail & Related papers  (2021-07-26T08:37:23Z)
• Fed-Focal Loss for imbalanced data classification in Federated Learning [2.2172881631608456]
  Federated Learning has a central server coordinating the training of a model on a network of devices. One of the challenges is variable training performance when the dataset has a class imbalance. We propose to address the class imbalance by reshaping cross-entropy loss such that it down-weights the loss assigned to well-classified examples along the lines of focal loss.
  arXiv  Detail & Related papers  (2020-11-12T09:52:14Z)
• Learnable Bernoulli Dropout for Bayesian Deep Learning [53.79615543862426]
  Learnable Bernoulli dropout (LBD) is a new model-agnostic dropout scheme that considers the dropout rates as parameters jointly optimized with other model parameters. LBD leads to improved accuracy and uncertainty estimates in image classification and semantic segmentation.
  arXiv  Detail & Related papers  (2020-02-12T18:57:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.