DropMix: Better Graph Contrastive Learning with Harder Negative Samples

• URL: http://arxiv.org/abs/2310.09764v1
• Date: Sun, 15 Oct 2023 07:45:30 GMT
• Title: DropMix: Better Graph Contrastive Learning with Harder Negative Samples
• Authors: Yueqi Ma, Minjie Chen, Xiang Li
• Abstract summary: Mixup has been introduced to synthesize hard negative samples in graph contrastive learning (GCL) We propose a novel method DropMix to synthesize harder negative samples.
• Score: 6.242575753642188
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While generating better negative samples for contrastive learning has been widely studied in the areas of CV and NLP, very few work has focused on graph-structured data. Recently, Mixup has been introduced to synthesize hard negative samples in graph contrastive learning (GCL). However, due to the unsupervised learning nature of GCL, without the help of soft labels, directly mixing representations of samples could inadvertently lead to the information loss of the original hard negative and further adversely affect the quality of the newly generated harder negative. To address the problem, in this paper, we propose a novel method DropMix to synthesize harder negative samples, which consists of two main steps. Specifically, we first select some hard negative samples by measuring their hardness from both local and global views in the graph simultaneously. After that, we mix hard negatives only on partial representation dimensions to generate harder ones and decrease the information loss caused by Mixup. We conduct extensive experiments to verify the effectiveness of DropMix on six benchmark datasets. Our results show that our method can lead to better GCL performance. Our data and codes are publicly available at https://github.com/Mayueq/DropMix-Code.

Related papers

• Synthetic Hard Negative Samples for Contrastive Learning [8.776888865665024]
  This paper proposes a novel feature-level method, namely sampling synthetic hard negative samples for contrastive learning (SSCL) We generate more and harder negative samples by mixing negative samples, and then sample them by controlling the contrast of anchor sample with the other negative samples. Our proposed method improves the classification performance on different image datasets and can be readily integrated into existing methods.
  arXiv  Detail & Related papers  (2023-04-06T09:54:35Z)
• DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification [56.817386699291305]
  This paper proposes a simple yet effective data augmentation approach termed DoubleMix. DoubleMix first generates several perturbed samples for each training data. It then uses the perturbed data and original data to carry out a two-step in the hidden space of neural models.
  arXiv  Detail & Related papers  (2022-09-12T15:01:04Z)
• Generating Counterfactual Hard Negative Samples for Graph Contrastive Learning [22.200011046576716]
  Graph contrastive learning is a powerful tool for unsupervised graph representation learning. Recent works usually sample negative samples from the same training batch with the positive samples, or from an external irrelevant graph. We propose a novel method to utilize textbfCounterfactual mechanism to generate artificial hard negative samples for textbfContrastive learning.
  arXiv  Detail & Related papers  (2022-07-01T02:19:59Z)
• MixKG: Mixing for harder negative samples in knowledge graph [33.4379457065033]
  Knowledge graph embedding(KGE) aims to represent entities and relations into low-dimensional vectors for many real-world applications. We introduce an inexpensive but effective method called MixKG to generate harder negative samples for knowledge graphs. Experiments on two public datasets and four classical KGE methods show MixKG is superior to previous negative sampling algorithms.
  arXiv  Detail & Related papers  (2022-02-19T13:31:06Z)
• Saliency Grafting: Innocuous Attribution-Guided Mixup with Calibrated Label Mixing [104.630875328668]
  Mixup scheme suggests mixing a pair of samples to create an augmented training sample. We present a novel, yet simple Mixup-variant that captures the best of both worlds.
  arXiv  Detail & Related papers  (2021-12-16T11:27:48Z)
• An Empirical Study of the Effects of Sample-Mixing Methods for Efficient Training of Generative Adversarial Networks [0.0]
  It is well-known that training of generative adversarial networks (GANs) requires huge iterations before the generator's providing good-quality samples. We investigated the effect of sample mixing methods, that is, Mixup, CutMix, and SRMix, to alleviate this problem.
  arXiv  Detail & Related papers  (2021-04-08T06:40:23Z)
• ReMix: Towards Image-to-Image Translation with Limited Data [154.71724970593036]
  We propose a data augmentation method (ReMix) to tackle this issue. We interpolate training samples at the feature level and propose a novel content loss based on the perceptual relations among samples. The proposed approach effectively reduces the ambiguity of generation and renders content-preserving results.
  arXiv  Detail & Related papers  (2021-03-31T06:24:10Z)
• Doubly Contrastive Deep Clustering [135.7001508427597]
  We present a novel Doubly Contrastive Deep Clustering (DCDC) framework, which constructs contrastive loss over both sample and class views. Specifically, for the sample view, we set the class distribution of the original sample and its augmented version as positive sample pairs. For the class view, we build the positive and negative pairs from the sample distribution of the class. In this way, two contrastive losses successfully constrain the clustering results of mini-batch samples in both sample and class level.
  arXiv  Detail & Related papers  (2021-03-09T15:15:32Z)
• Contrastive Learning with Hard Negative Samples [80.12117639845678]
  We develop a new family of unsupervised sampling methods for selecting hard negative samples. A limiting case of this sampling results in a representation that tightly clusters each class, and pushes different classes as far apart as possible. The proposed method improves downstream performance across multiple modalities, requires only few additional lines of code to implement, and introduces no computational overhead.
  arXiv  Detail & Related papers  (2020-10-09T14:18:53Z)
• SCE: Scalable Network Embedding from Sparsest Cut [20.08464038805681]
  Large-scale network embedding is to learn a latent representation for each node in an unsupervised manner. A key of success to such contrastive learning methods is how to draw positive and negative samples. In this paper, we propose SCE for unsupervised network embedding only using negative samples for training.
  arXiv  Detail & Related papers  (2020-06-30T03:18:15Z)
• DivideMix: Learning with Noisy Labels as Semi-supervised Learning [111.03364864022261]
  We propose DivideMix, a framework for learning with noisy labels. Experiments on multiple benchmark datasets demonstrate substantial improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-02-18T06:20:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.