A Unified Contrastive Loss for Self-Training

• URL: http://arxiv.org/abs/2409.07292v1
• Date: Wed, 11 Sep 2024 14:22:41 GMT
• Title: A Unified Contrastive Loss for Self-Training
• Authors: Aurelien Gauffre, Julien Horvat, Massih-Reza Amini,
• Abstract summary: Self-training methods have proven to be effective in exploiting abundant unlabeled data in semi-supervised learning. We propose a general framework to enhance self-training methods, which replaces all instances of CE losses with a unique contrastive loss. Our framework results in significant performance improvements across three different datasets with a limited number of labeled data.
• Score: 3.3454373538792552
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-training methods have proven to be effective in exploiting abundant unlabeled data in semi-supervised learning, particularly when labeled data is scarce. While many of these approaches rely on a cross-entropy loss function (CE), recent advances have shown that the supervised contrastive loss function (SupCon) can be more effective. Additionally, unsupervised contrastive learning approaches have also been shown to capture high quality data representations in the unsupervised setting. To benefit from these advantages in a semi-supervised setting, we propose a general framework to enhance self-training methods, which replaces all instances of CE losses with a unique contrastive loss. By using class prototypes, which are a set of class-wise trainable parameters, we recover the probability distributions of the CE setting and show a theoretical equivalence with it. Our framework, when applied to popular self-training methods, results in significant performance improvements across three different datasets with a limited number of labeled data. Additionally, we demonstrate further improvements in convergence speed, transfer ability, and hyperparameter stability. The code is available at \url{https://github.com/AurelienGauffre/semisupcon/}.

Related papers

• EnsLoss: Stochastic Calibrated Loss Ensembles for Preventing Overfitting in Classification [1.3778851745408134]
  We propose a novel ensemble method, namely EnsLoss, to combine loss functions within the Empirical risk minimization framework. We first transform the CC conditions of losses into loss-derivatives, thereby bypassing the need for explicit loss functions. We theoretically establish the statistical consistency of our approach and provide insights into its benefits.
  arXiv  Detail & Related papers  (2024-09-02T02:40:42Z)
• Binary Classification with Confidence Difference [100.08818204756093]
  This paper delves into a novel weakly supervised binary classification problem called confidence-difference (ConfDiff) classification. We propose a risk-consistent approach to tackle this problem and show that the estimation error bound the optimal convergence rate. We also introduce a risk correction approach to mitigate overfitting problems, whose consistency and convergence rate are also proven.
  arXiv  Detail & Related papers  (2023-10-09T11:44:50Z)
• Tuned Contrastive Learning [77.67209954169593]
  We propose a novel contrastive loss function -- Tuned Contrastive Learning (TCL) loss. TCL generalizes to multiple positives and negatives in a batch and offers parameters to tune and improve the gradient responses from hard positives and hard negatives. We show how to extend TCL to self-supervised setting and empirically compare it with various SOTA self-supervised learning methods.
  arXiv  Detail & Related papers  (2023-05-18T03:26:37Z)
• Positive-Negative Equal Contrastive Loss for Semantic Segmentation [8.664491798389662]
  Previous works commonly design plug-and-play modules and structural losses to effectively extract and aggregate the global context. We propose Positive-Negative Equal contrastive loss (PNE loss), which increases the latent impact of positive embedding on the anchor and treats the positive as well as negative sample pairs equally. We conduct comprehensive experiments and achieve state-of-the-art performance on two benchmark datasets.
  arXiv  Detail & Related papers  (2022-07-04T13:51:29Z)
• Distributed Adversarial Training to Robustify Deep Neural Networks at Scale [100.19539096465101]
  Current deep neural networks (DNNs) are vulnerable to adversarial attacks, where adversarial perturbations to the inputs can change or manipulate classification. To defend against such attacks, an effective approach, known as adversarial training (AT), has been shown to mitigate robust training. We propose a large-batch adversarial training framework implemented over multiple machines.
  arXiv  Detail & Related papers  (2022-06-13T15:39:43Z)
• Adversarial Dual-Student with Differentiable Spatial Warping for Semi-Supervised Semantic Segmentation [70.2166826794421]
  We propose a differentiable geometric warping to conduct unsupervised data augmentation. We also propose a novel adversarial dual-student framework to improve the Mean-Teacher. Our solution significantly improves the performance and state-of-the-art results are achieved on both datasets.
  arXiv  Detail & Related papers  (2022-03-05T17:36:17Z)
• Guided Point Contrastive Learning for Semi-supervised Point Cloud Semantic Segmentation [90.2445084743881]
  We present a method for semi-supervised point cloud semantic segmentation to adopt unlabeled point clouds in training to boost the model performance. Inspired by the recent contrastive loss in self-supervised tasks, we propose the guided point contrastive loss to enhance the feature representation and model generalization ability.
  arXiv  Detail & Related papers  (2021-10-15T16:38:54Z)
• Robust Disentanglement of a Few Factors at a Time [5.156484100374058]
  We introduce population-based training (PBT) for improving consistency in training variational autoencoders (VAEs) We then use Unsupervised Disentanglement Ranking (UDR) as an unsupervised to score models in our PBT-VAE training and show how models trained this way tend to consistently disentangle only a subset of the generative factors. We show striking improvement in state-of-the-art unsupervised disentanglement performance and robustness across multiple datasets and metrics.
  arXiv  Detail & Related papers  (2020-10-26T12:34:23Z)
• Robust Pre-Training by Adversarial Contrastive Learning [120.33706897927391]
  Recent work has shown that, when integrated with adversarial training, self-supervised pre-training can lead to state-of-the-art robustness. We improve robustness-aware self-supervised pre-training by learning representations consistent under both data augmentations and adversarial perturbations.
  arXiv  Detail & Related papers  (2020-10-26T04:44:43Z)
• Supervised Contrastive Learning [42.27949000093086]
  We extend the self-supervised batch contrastive approach to the fully-supervised setting. We analyze two possible versions of the supervised contrastive (SupCon) loss, identifying the best-performing formulation of the loss. On ResNet-200, we achieve top-1 accuracy of 81.4% on the ImageNet dataset, which is 0.8% above the best number reported for this architecture.
  arXiv  Detail & Related papers  (2020-04-23T17:58:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.