Rethinking Samples Selection for Contrastive Learning: Mining of Potential Samples

• URL: http://arxiv.org/abs/2311.00358v1
• Date: Wed, 1 Nov 2023 08:08:06 GMT
• Title: Rethinking Samples Selection for Contrastive Learning: Mining of Potential Samples
• Authors: Hengkui Dong, Xianzhong Long, Yun Li
• Abstract summary: Contrastive learning predicts whether two images belong to the same category by training a model to make their feature representations as close or as far away as possible. We take into account both positive and negative samples, and mining potential samples from two aspects. Our method achieves 88.57%, 61.10%, and 36.69% top-1 accuracy on CIFAR10, CIFAR100, and TinyImagenet, respectively.
• Score: 5.586563813796839
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive learning predicts whether two images belong to the same category by training a model to make their feature representations as close or as far away as possible. In this paper, we rethink how to mine samples in contrastive learning, unlike other methods, our approach is more comprehensive, taking into account both positive and negative samples, and mining potential samples from two aspects: First, for positive samples, we consider both the augmented sample views obtained by data augmentation and the mined sample views through data mining. Then, we weight and combine them using both soft and hard weighting strategies. Second, considering the existence of uninformative negative samples and false negative samples in the negative samples, we analyze the negative samples from the gradient perspective and finally mine negative samples that are neither too hard nor too easy as potential negative samples, i.e., those negative samples that are close to positive samples. The experiments show the obvious advantages of our method compared with some traditional self-supervised methods. Our method achieves 88.57%, 61.10%, and 36.69% top-1 accuracy on CIFAR10, CIFAR100, and TinyImagenet, respectively.

Related papers

• From Overfitting to Robustness: Quantity, Quality, and Variety Oriented Negative Sample Selection in Graph Contrastive Learning [38.87932592059369]
  Graph contrastive learning (GCL) aims to contrast positive-negative counterparts to learn the node embeddings. The variation, quantity, and quality of negative samples compared to positive samples play crucial roles in learning meaningful embeddings for node classification downstream tasks. This study proposes a novel Cumulative Sample Selection (CSS) algorithm by comprehensively considering negative samples' quality, variations, and quantity.
  arXiv  Detail & Related papers  (2024-06-21T10:47:26Z)
• Your Negative May not Be True Negative: Boosting Image-Text Matching with False Negative Elimination [62.18768931714238]
  We propose a novel False Negative Elimination (FNE) strategy to select negatives via sampling. The results demonstrate the superiority of our proposed false negative elimination strategy.
  arXiv  Detail & Related papers  (2023-08-08T16:31:43Z)
• Synthetic Hard Negative Samples for Contrastive Learning [8.776888865665024]
  This paper proposes a novel feature-level method, namely sampling synthetic hard negative samples for contrastive learning (SSCL) We generate more and harder negative samples by mixing negative samples, and then sample them by controlling the contrast of anchor sample with the other negative samples. Our proposed method improves the classification performance on different image datasets and can be readily integrated into existing methods.
  arXiv  Detail & Related papers  (2023-04-06T09:54:35Z)
• SimANS: Simple Ambiguous Negatives Sampling for Dense Text Retrieval [126.22182758461244]
  We show that according to the measured relevance scores, the negatives ranked around the positives are generally more informative and less likely to be false negatives. We propose a simple ambiguous negatives sampling method, SimANS, which incorporates a new sampling probability distribution to sample more ambiguous negatives.
  arXiv  Detail & Related papers  (2022-10-21T07:18:05Z)
• Generating Counterfactual Hard Negative Samples for Graph Contrastive Learning [22.200011046576716]
  Graph contrastive learning is a powerful tool for unsupervised graph representation learning. Recent works usually sample negative samples from the same training batch with the positive samples, or from an external irrelevant graph. We propose a novel method to utilize textbfCounterfactual mechanism to generate artificial hard negative samples for textbfContrastive learning.
  arXiv  Detail & Related papers  (2022-07-01T02:19:59Z)
• Rethinking InfoNCE: How Many Negative Samples Do You Need? [54.146208195806636]
  We study how many negative samples are optimal for InfoNCE in different scenarios via a semi-quantitative theoretical framework. We estimate the optimal negative sampling ratio using the $K$ value that maximizes the training effectiveness function.
  arXiv  Detail & Related papers  (2021-05-27T08:38:29Z)
• Contrastive Attraction and Contrastive Repulsion for Representation Learning [131.72147978462348]
  Contrastive learning (CL) methods learn data representations in a self-supervision manner, where the encoder contrasts each positive sample over multiple negative samples. Recent CL methods have achieved promising results when pretrained on large-scale datasets, such as ImageNet. We propose a doubly CL strategy that separately compares positive and negative samples within their own groups, and then proceeds with a contrast between positive and negative groups.
  arXiv  Detail & Related papers  (2021-05-08T17:25:08Z)
• Doubly Contrastive Deep Clustering [135.7001508427597]
  We present a novel Doubly Contrastive Deep Clustering (DCDC) framework, which constructs contrastive loss over both sample and class views. Specifically, for the sample view, we set the class distribution of the original sample and its augmented version as positive sample pairs. For the class view, we build the positive and negative pairs from the sample distribution of the class. In this way, two contrastive losses successfully constrain the clustering results of mini-batch samples in both sample and class level.
  arXiv  Detail & Related papers  (2021-03-09T15:15:32Z)
• Contrastive Learning with Hard Negative Samples [80.12117639845678]
  We develop a new family of unsupervised sampling methods for selecting hard negative samples. A limiting case of this sampling results in a representation that tightly clusters each class, and pushes different classes as far apart as possible. The proposed method improves downstream performance across multiple modalities, requires only few additional lines of code to implement, and introduces no computational overhead.
  arXiv  Detail & Related papers  (2020-10-09T14:18:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.