About contrastive unsupervised representation learning for classification and its convergence

• URL: http://arxiv.org/abs/2012.01064v1
• Date: Wed, 2 Dec 2020 10:08:57 GMT
• Title: About contrastive unsupervised representation learning for classification and its convergence
• Authors: Ibrahim Merad and Yiyang Yu and Emmanuel Bacry and St\'ephane Ga\"iffas
• Abstract summary: We build a theoretical framework around contrastive learning which guarantees for its performance can be proven. We provide extensions of these results to training with multiple negative samples and for multiway classification. We also provide convergence guarantees for the minimization of the contrastive training error with gradient descent of an overparametrized deep neural encoder.
• Score: 2.6782615615913343
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive representation learning has been recently proved to be very efficient for self-supervised training. These methods have been successfully used to train encoders which perform comparably to supervised training on downstream classification tasks. A few works have started to build a theoretical framework around contrastive learning in which guarantees for its performance can be proven. We provide extensions of these results to training with multiple negative samples and for multiway classification. Furthermore, we provide convergence guarantees for the minimization of the contrastive training error with gradient descent of an overparametrized deep neural encoder, and provide some numerical experiments that complement our theoretical findings

Related papers

• Adversarial Training Can Provably Improve Robustness: Theoretical Analysis of Feature Learning Process Under Structured Data [38.44734564565478]
  We provide a theoretical understanding of adversarial examples and adversarial training algorithms from the perspective of feature learning theory. We show that the adversarial training method can provably strengthen the robust feature learning and suppress the non-robust feature learning.
  arXiv  Detail & Related papers  (2024-10-11T03:59:49Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap [64.60460828425502]
  We propose a new guarantee on the downstream performance of contrastive learning. Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations. We propose an unsupervised model selection metric ARC that aligns well with downstream accuracy.
  arXiv  Detail & Related papers  (2022-03-25T05:36:26Z)
• Contrastive Learning for Fair Representations [50.95604482330149]
  Trained classification models can unintentionally lead to biased representations and predictions. Existing debiasing methods for classification models, such as adversarial training, are often expensive to train and difficult to optimise. We propose a method for mitigating bias by incorporating contrastive learning, in which instances sharing the same class label are encouraged to have similar representations.
  arXiv  Detail & Related papers  (2021-09-22T10:47:51Z)
• Improving Music Performance Assessment with Contrastive Learning [78.8942067357231]
  This study investigates contrastive learning as a potential method to improve existing MPA systems. We introduce a weighted contrastive loss suitable for regression tasks applied to a convolutional neural network. Our results show that contrastive-based methods are able to match and exceed SoTA performance for MPA regression tasks.
  arXiv  Detail & Related papers  (2021-08-03T19:24:25Z)
• Solving Inefficiency of Self-supervised Representation Learning [87.30876679780532]
  Existing contrastive learning methods suffer from very low learning efficiency. Under-clustering and over-clustering problems are major obstacles to learning efficiency. We propose a novel self-supervised learning framework using a median triplet loss.
  arXiv  Detail & Related papers  (2021-04-18T07:47:10Z)
• Unsupervised Class-Incremental Learning Through Confusion [0.4604003661048266]
  We introduce a novelty detection method that leverages network confusion caused by training incoming data as a new class. We found that incorporating a class-imbalance during this detection method substantially enhances performance.
  arXiv  Detail & Related papers  (2021-04-09T15:58:43Z)
• Accumulated Decoupled Learning: Mitigating Gradient Staleness in Inter-Layer Model Parallelization [16.02377434191239]
  We propose an accumulated decoupled learning (ADL) which incorporates the gradient accumulation technique to mitigate the stale gradient effect. We prove that the proposed method can converge to critical points, i.e., the gradients converge to 0, in spite of its asynchronous nature. The ADL is shown to outperform several state-of-the-arts in the classification tasks, and is the fastest among the compared methods.
  arXiv  Detail & Related papers  (2020-12-03T11:52:55Z)
• Learning Diverse Representations for Fast Adaptation to Distribution Shift [78.83747601814669]
  We present a method for learning multiple models, incorporating an objective that pressures each to learn a distinct way to solve the task. We demonstrate our framework's ability to facilitate rapid adaptation to distribution shift.
  arXiv  Detail & Related papers  (2020-06-12T12:23:50Z)
• Convergence of End-to-End Training in Deep Unsupervised Contrastive Learning [3.8073142980733]
  Unsupervised contrastive learning has proven to be a powerful method for learning representations from unlabeled data. This study provides theoretical insights into the practical success of these unsupervised methods.
  arXiv  Detail & Related papers  (2020-02-17T14:35:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.