Investigating the Role of Negatives in Contrastive Representation Learning

• URL: http://arxiv.org/abs/2106.09943v1
• Date: Fri, 18 Jun 2021 06:44:16 GMT
• Title: Investigating the Role of Negatives in Contrastive Representation Learning
• Authors: Jordan T. Ash, Surbhi Goel, Akshay Krishnamurthy and Dipendra Misra
• Abstract summary: Noise contrastive learning is a popular technique for unsupervised representation learning. We focus on disambiguating the role of one of these parameters: the number of negative examples. We find that the results broadly agree with our theory, while our vision experiments are murkier with performance sometimes even being insensitive to the number of negatives.
• Score: 59.30700308648194
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Noise contrastive learning is a popular technique for unsupervised representation learning. In this approach, a representation is obtained via reduction to supervised learning, where given a notion of semantic similarity, the learner tries to distinguish a similar (positive) example from a collection of random (negative) examples. The success of modern contrastive learning pipelines relies on many parameters such as the choice of data augmentation, the number of negative examples, and the batch size; however, there is limited understanding as to how these parameters interact and affect downstream performance. We focus on disambiguating the role of one of these parameters: the number of negative examples. Theoretically, we show the existence of a collision-coverage trade-off suggesting that the optimal number of negative examples should scale with the number of underlying concepts in the data. Empirically, we scrutinize the role of the number of negatives in both NLP and vision tasks. In the NLP task, we find that the results broadly agree with our theory, while our vision experiments are murkier with performance sometimes even being insensitive to the number of negatives. We discuss plausible explanations for this behavior and suggest future directions to better align theory and practice.

Related papers

• KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning [31.139620652818838]
  We propose KDMCSE, a novel approach that enhances the discrimination and generalizability of multimodal representation. We also introduce a new contrastive objective, AdapACSE, that enhances the discriminative representation by strengthening the margin within the angular space.
  arXiv  Detail & Related papers  (2024-03-26T08:32:39Z)
• Clustering-Aware Negative Sampling for Unsupervised Sentence Representation [24.15096466098421]
  ClusterNS is a novel method that incorporates cluster information into contrastive learning for unsupervised sentence representation learning. We apply a modified K-means clustering algorithm to supply hard negatives and recognize in-batch false negatives during training.
  arXiv  Detail & Related papers  (2023-05-17T02:06:47Z)
• Understanding Contrastive Learning Requires Incorporating Inductive Biases [64.56006519908213]
  Recent attempts to theoretically explain the success of contrastive learning on downstream tasks prove guarantees depending on properties of em augmentations and the value of em contrastive loss of representations. We demonstrate that such analyses ignore em inductive biases of the function class and training algorithm, even em provably leading to vacuous guarantees in some settings.
  arXiv  Detail & Related papers  (2022-02-28T18:59:20Z)
• Incremental False Negative Detection for Contrastive Learning [95.68120675114878]
  We introduce a novel incremental false negative detection for self-supervised contrastive learning. During contrastive learning, we discuss two strategies to explicitly remove the detected false negatives. Our proposed method outperforms other self-supervised contrastive learning frameworks on multiple benchmarks within a limited compute.
  arXiv  Detail & Related papers  (2021-06-07T15:29:14Z)
• Solving Inefficiency of Self-supervised Representation Learning [87.30876679780532]
  Existing contrastive learning methods suffer from very low learning efficiency. Under-clustering and over-clustering problems are major obstacles to learning efficiency. We propose a novel self-supervised learning framework using a median triplet loss.
  arXiv  Detail & Related papers  (2021-04-18T07:47:10Z)
• Understanding Hard Negatives in Noise Contrastive Estimation [21.602701327267905]
  We develop analytical tools to understand the role of hard negatives. We derive a general form of the score function that unifies various architectures used in text retrieval.
  arXiv  Detail & Related papers  (2021-04-13T14:42:41Z)
• Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learning [29.583194697391253]
  Self-supervised representation learning commonly uses more negative samples than the number of supervised classes in practice. We theoretically explain this empirical result regarding negative samples. We empirically confirm our analysis by conducting numerical experiments on CIFAR-10/100 datasets.
  arXiv  Detail & Related papers  (2021-02-13T05:46:33Z)
• AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversaries [55.059844800514774]
  We propose an Adrial Contrastive (AdCo) model to train representations that are hard to discriminate against positive queries. Experiment results demonstrate the proposed Adrial Contrastive (AdCo) model achieves superior performances.
  arXiv  Detail & Related papers  (2020-11-17T05:45:46Z)
• Contrastive Learning with Hard Negative Samples [80.12117639845678]
  We develop a new family of unsupervised sampling methods for selecting hard negative samples. A limiting case of this sampling results in a representation that tightly clusters each class, and pushes different classes as far apart as possible. The proposed method improves downstream performance across multiple modalities, requires only few additional lines of code to implement, and introduces no computational overhead.
  arXiv  Detail & Related papers  (2020-10-09T14:18:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.