Tuned Contrastive Learning

• URL: http://arxiv.org/abs/2305.10675v2
• Date: Tue, 30 May 2023 05:00:37 GMT
• Title: Tuned Contrastive Learning
• Authors: Chaitanya Animesh, Manmohan Chandraker
• Abstract summary: We propose a novel contrastive loss function -- Tuned Contrastive Learning (TCL) loss. TCL generalizes to multiple positives and negatives in a batch and offers parameters to tune and improve the gradient responses from hard positives and hard negatives. We show how to extend TCL to self-supervised setting and empirically compare it with various SOTA self-supervised learning methods.
• Score: 77.67209954169593
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent times, contrastive learning based loss functions have become increasingly popular for visual self-supervised representation learning owing to their state-of-the-art (SOTA) performance. Most of the modern contrastive learning methods generalize only to one positive and multiple negatives per anchor. A recent state-of-the-art, supervised contrastive (SupCon) loss, extends self-supervised contrastive learning to supervised setting by generalizing to multiple positives and negatives in a batch and improves upon the cross-entropy loss. In this paper, we propose a novel contrastive loss function -- Tuned Contrastive Learning (TCL) loss, that generalizes to multiple positives and negatives in a batch and offers parameters to tune and improve the gradient responses from hard positives and hard negatives. We provide theoretical analysis of our loss function's gradient response and show mathematically how it is better than that of SupCon loss. We empirically compare our loss function with SupCon loss and cross-entropy loss in supervised setting on multiple classification-task datasets to show its effectiveness. We also show the stability of our loss function to a range of hyper-parameter settings. Unlike SupCon loss which is only applied to supervised setting, we show how to extend TCL to self-supervised setting and empirically compare it with various SOTA self-supervised learning methods. Hence, we show that TCL loss achieves performance on par with SOTA methods in both supervised and self-supervised settings.

Related papers

• A Unified Contrastive Loss for Self-Training [3.3454373538792552]
  Self-training methods have proven to be effective in exploiting abundant unlabeled data in semi-supervised learning. We propose a general framework to enhance self-training methods, which replaces all instances of CE losses with a unique contrastive loss. Our framework results in significant performance improvements across three different datasets with a limited number of labeled data.
  arXiv  Detail & Related papers  (2024-09-11T14:22:41Z)
• Understanding Collapse in Non-Contrastive Learning [122.2499276246997]
  We show that SimSiam representations undergo partial dimensional collapse if the model is too small relative to the dataset size. We propose a metric to measure the degree of this collapse and show that it can be used to forecast the downstream task performance without any fine-tuning or labels.
  arXiv  Detail & Related papers  (2022-09-29T17:59:55Z)
• Positive-Negative Equal Contrastive Loss for Semantic Segmentation [8.664491798389662]
  Previous works commonly design plug-and-play modules and structural losses to effectively extract and aggregate the global context. We propose Positive-Negative Equal contrastive loss (PNE loss), which increases the latent impact of positive embedding on the anchor and treats the positive as well as negative sample pairs equally. We conduct comprehensive experiments and achieve state-of-the-art performance on two benchmark datasets.
  arXiv  Detail & Related papers  (2022-07-04T13:51:29Z)
• Do Lessons from Metric Learning Generalize to Image-Caption Retrieval? [67.45267657995748]
  The triplet loss with semi-hard negatives has become the de facto choice for image-caption retrieval (ICR) methods that are optimized from scratch. Recent progress in metric learning has given rise to new loss functions that outperform the triplet loss on tasks such as image retrieval and representation learning. We ask whether these findings generalize to the setting of ICR by comparing three loss functions on two ICR methods.
  arXiv  Detail & Related papers  (2022-02-14T15:18:00Z)
• Mixing between the Cross Entropy and the Expectation Loss Terms [89.30385901335323]
  Cross entropy loss tends to focus on hard to classify samples during training. We show that adding to the optimization goal the expectation loss helps the network to achieve better accuracy. Our experiments show that the new training protocol improves performance across a diverse set of classification domains.
  arXiv  Detail & Related papers  (2021-09-12T23:14:06Z)
• Investigating the Role of Negatives in Contrastive Representation Learning [59.30700308648194]
  Noise contrastive learning is a popular technique for unsupervised representation learning. We focus on disambiguating the role of one of these parameters: the number of negative examples. We find that the results broadly agree with our theory, while our vision experiments are murkier with performance sometimes even being insensitive to the number of negatives.
  arXiv  Detail & Related papers  (2021-06-18T06:44:16Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)
• Class Interference Regularization [7.248447600071719]
  Contrastive losses yield state-of-the-art performance for person re-identification, face verification and few shot learning. We propose a novel, simple and effective regularization technique, the Class Interference Regularization (CIR) CIR perturbs the output features by randomly moving them towards the average embeddings of the negative classes.
  arXiv  Detail & Related papers  (2020-09-04T21:03:32Z)
• Supervised Contrastive Learning [42.27949000093086]
  We extend the self-supervised batch contrastive approach to the fully-supervised setting. We analyze two possible versions of the supervised contrastive (SupCon) loss, identifying the best-performing formulation of the loss. On ResNet-200, we achieve top-1 accuracy of 81.4% on the ImageNet dataset, which is 0.8% above the best number reported for this architecture.
  arXiv  Detail & Related papers  (2020-04-23T17:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.