BatchSampler: Sampling Mini-Batches for Contrastive Learning in Vision, Language, and Graphs

• URL: http://arxiv.org/abs/2306.03355v1
• Date: Tue, 6 Jun 2023 02:13:27 GMT
• Title: BatchSampler: Sampling Mini-Batches for Contrastive Learning in Vision, Language, and Graphs
• Authors: Zhen Yang, Tinglin Huang, Ming Ding, Yuxiao Dong, Rex Ying, Yukuo Cen, Yangliao Geng, and Jie Tang
• Abstract summary: In-Batch contrastive learning is a state-of-the-art self-supervised method that brings semantically-similar instances close. Recent studies aim to improve performance by sampling hard negatives textitwithin the current mini-batch. We present BatchSampler to sample mini-batches of hard-to-distinguish (i.e., hard and true negatives to each other) instances.
• Score: 37.378865860897285
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In-Batch contrastive learning is a state-of-the-art self-supervised method that brings semantically-similar instances close while pushing dissimilar instances apart within a mini-batch. Its key to success is the negative sharing strategy, in which every instance serves as a negative for the others within the mini-batch. Recent studies aim to improve performance by sampling hard negatives \textit{within the current mini-batch}, whose quality is bounded by the mini-batch itself. In this work, we propose to improve contrastive learning by sampling mini-batches from the input data. We present BatchSampler\footnote{The code is available at \url{https://github.com/THUDM/BatchSampler}} to sample mini-batches of hard-to-distinguish (i.e., hard and true negatives to each other) instances. To make each mini-batch have fewer false negatives, we design the proximity graph of randomly-selected instances. To form the mini-batch, we leverage random walk with restart on the proximity graph to help sample hard-to-distinguish instances. BatchSampler is a simple and general technique that can be directly plugged into existing contrastive learning models in vision, language, and graphs. Extensive experiments on datasets of three modalities show that BatchSampler can consistently improve the performance of powerful contrastive models, as shown by significant improvements of SimCLR on ImageNet-100, SimCSE on STS (language), and GraphCL and MVGRL on graph datasets.

Related papers

• $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs [62.565573316667276]
  We develop an objective that encodes how a sample relates to others. We train vision models based on similarities in class or text caption descriptions. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $16.8%$ on ImageNet and $18.1%$ on ImageNet Real.
  arXiv  Detail & Related papers  (2024-07-25T15:38:16Z)
• Mini-Batch Optimization of Contrastive Loss [13.730030395850358]
  We show that mini-batch optimization is equivalent to full-batch optimization if and only if all $binomNB$ mini-batches are selected. We then demonstrate that utilizing high-loss mini-batches can speed up SGD convergence and propose a spectral clustering-based approach for identifying these high-loss mini-batches.
  arXiv  Detail & Related papers  (2023-07-12T04:23:26Z)
• MSVQ: Self-Supervised Learning with Multiple Sample Views and Queues [10.327408694770709]
  We propose a new simple framework, namely Multiple Sample Views and Queues (MSVQ) We jointly construct three soft labels on-the-fly by utilizing two complementary and symmetric approaches. Let the student network mimic the similarity relationships between the samples, thus giving the student network a more flexible ability to identify false negative samples in the dataset.
  arXiv  Detail & Related papers  (2023-05-09T12:05:14Z)
• Rethinking Batch Sample Relationships for Data Representation: A Batch-Graph Transformer based Approach [16.757917001089762]
  We design a simple yet flexible Batch-Graph Transformer (BGFormer) for mini-batch sample representations. It deeply captures the relationships of image samples from both visual and semantic perspectives. Extensive experiments on four popular datasets demonstrate the effectiveness of the proposed model.
  arXiv  Detail & Related papers  (2022-11-19T08:46:50Z)
• BatchFormer: Learning to Explore Sample Relationships for Robust Representation Learning [93.38239238988719]
  We propose to enable deep neural networks with the ability to learn the sample relationships from each mini-batch. BatchFormer is applied into the batch dimension of each mini-batch to implicitly explore sample relationships during training. We perform extensive experiments on over ten datasets and the proposed method achieves significant improvements on different data scarcity applications.
  arXiv  Detail & Related papers  (2022-03-03T05:31:33Z)
• Bag of Instances Aggregation Boosts Self-supervised Learning [122.61914701794296]
  We propose a simple but effective distillation strategy for unsupervised learning. Our method, termed as BINGO, targets at transferring the relationship learned by the teacher to the student. BINGO achieves new state-of-the-art performance on small scale models.
  arXiv  Detail & Related papers  (2021-07-04T17:33:59Z)
• Graph Sampling Based Deep Metric Learning for Generalizable Person Re-Identification [114.56752624945142]
  We argue that the most popular random sampling method, the well-known PK sampler, is not informative and efficient for deep metric learning. We propose an efficient mini batch sampling method called Graph Sampling (GS) for large-scale metric learning.
  arXiv  Detail & Related papers  (2021-04-04T06:44:15Z)
• Doubly Contrastive Deep Clustering [135.7001508427597]
  We present a novel Doubly Contrastive Deep Clustering (DCDC) framework, which constructs contrastive loss over both sample and class views. Specifically, for the sample view, we set the class distribution of the original sample and its augmented version as positive sample pairs. For the class view, we build the positive and negative pairs from the sample distribution of the class. In this way, two contrastive losses successfully constrain the clustering results of mini-batch samples in both sample and class level.
  arXiv  Detail & Related papers  (2021-03-09T15:15:32Z)
• Exploring Effects of Random Walk Based Minibatch Selection Policy on Knowledge Graph Completion [11.484811954887432]
  We propose a new random-walk based minibatch sampling technique for training KGC models. We find that our proposed method achieves state-of-the-art performance on the DB100K dataset.
  arXiv  Detail & Related papers  (2020-04-12T06:16:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.