Generalized Supervised Contrastive Learning

• URL: http://arxiv.org/abs/2206.00384v2
• Date: Sun, 21 May 2023 16:50:33 GMT
• Title: Generalized Supervised Contrastive Learning
• Authors: Jaewon Kim, Hyukjong Lee, Jooyoung Chang, Sang Min Park
• Abstract summary: We introduce a generalized supervised contrastive loss, which measures cross-entropy between label similarity and latent similarity. Compared to existing contrastive learning frameworks, we construct a tailored framework: the Generalized Supervised Contrastive Learning (GenSCL) GenSCL achieves a top-1 accuracy of 77.3% on ImageNet, a 4.1% improvement over traditional supervised contrastive learning.
• Score: 3.499054375230853
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With the recent promising results of contrastive learning in the self-supervised learning paradigm, supervised contrastive learning has successfully extended these contrastive approaches to supervised contexts, outperforming cross-entropy on various datasets. However, supervised contrastive learning inherently employs label information in a binary form--either positive or negative--using a one-hot target vector. This structure struggles to adapt to methods that exploit label information as a probability distribution, such as CutMix and knowledge distillation. In this paper, we introduce a generalized supervised contrastive loss, which measures cross-entropy between label similarity and latent similarity. This concept enhances the capabilities of supervised contrastive loss by fully utilizing the label distribution and enabling the adaptation of various existing techniques for training modern neural networks. Leveraging this generalized supervised contrastive loss, we construct a tailored framework: the Generalized Supervised Contrastive Learning (GenSCL). Compared to existing contrastive learning frameworks, GenSCL incorporates additional enhancements, including advanced image-based regularization techniques and an arbitrary teacher classifier. When applied to ResNet50 with the Momentum Contrast technique, GenSCL achieves a top-1 accuracy of 77.3% on ImageNet, a 4.1% relative improvement over traditional supervised contrastive learning. Moreover, our method establishes new state-of-the-art accuracies of 98.2% and 87.0% on CIFAR10 and CIFAR100 respectively when applied to ResNet50, marking the highest reported figures for this architecture.

Related papers

• Bayesian Learning-driven Prototypical Contrastive Loss for Class-Incremental Learning [42.14439854721613]
  We propose a prototypical network with a Bayesian learning-driven contrastive loss (BLCL) tailored specifically for class-incremental learning scenarios. Our approach dynamically adapts the balance between the cross-entropy and contrastive loss functions with a Bayesian learning technique.
  arXiv  Detail & Related papers  (2024-05-17T19:49:02Z)
• Weakly Supervised Contrastive Learning [68.47096022526927]
  We introduce a weakly supervised contrastive learning framework (WCL) to tackle this issue. WCL achieves 65% and 72% ImageNet Top-1 Accuracy using ResNet50, which is even higher than SimCLRv2 with ResNet101.
  arXiv  Detail & Related papers  (2021-10-10T12:03:52Z)
• To be Critical: Self-Calibrated Weakly Supervised Learning for Salient Object Detection [95.21700830273221]
  Weakly-supervised salient object detection (WSOD) aims to develop saliency models using image-level annotations. We propose a self-calibrated training strategy by explicitly establishing a mutual calibration loop between pseudo labels and network predictions. We prove that even a much smaller dataset with well-matched annotations can facilitate models to achieve better performance as well as generalizability.
  arXiv  Detail & Related papers  (2021-09-04T02:45:22Z)
• Semi-supervised Contrastive Learning with Similarity Co-calibration [72.38187308270135]
  We propose a novel training strategy, termed as Semi-supervised Contrastive Learning (SsCL) SsCL combines the well-known contrastive loss in self-supervised learning with the cross entropy loss in semi-supervised learning. We show that SsCL produces more discriminative representation and is beneficial to few shot learning.
  arXiv  Detail & Related papers  (2021-05-16T09:13:56Z)
• Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification [31.647639786095993]
  We propose a novel hybrid network structure composed of a supervised contrastive loss to learn image representations and a cross-entropy loss to learn classifiers. Experiments on three long-tailed classification datasets demonstrate the advantage of the proposed contrastive learning based hybrid networks in long-tailed classification.
  arXiv  Detail & Related papers  (2021-03-26T05:22:36Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)
• Hybrid Discriminative-Generative Training via Contrastive Learning [96.56164427726203]
  We show that through the perspective of hybrid discriminative-generative training of energy-based models we can make a direct connection between contrastive learning and supervised learning. We show our specific choice of approximation of the energy-based loss outperforms the existing practice in terms of classification accuracy of WideResNet on CIFAR-10 and CIFAR-100.
  arXiv  Detail & Related papers  (2020-07-17T15:50:34Z)
• A Simple Framework for Contrastive Learning of Visual Representations [116.37752766922407]
  This paper presents SimCLR: a simple framework for contrastive learning of visual representations. We show that composition of data augmentations plays a critical role in defining effective predictive tasks. We are able to considerably outperform previous methods for self-supervised and semi-supervised learning on ImageNet.
  arXiv  Detail & Related papers  (2020-02-13T18:50:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.