Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining

• URL: http://arxiv.org/abs/2505.11293v1
• Date: Fri, 16 May 2025 14:25:43 GMT
• Title: Breaking the Batch Barrier (B3) of Contrastive Learning via Smart Batch Mining
• Authors: Raghuveer Thirukovalluru, Rui Meng, Ye Liu, Karthikeyan K, Mingyi Su, Ping Nie, Semih Yavuz, Yingbo Zhou, Wenhu Chen, Bhuwan Dhingra,
• Abstract summary: 'Breaking the Batch Barrier' (B3) is a novel batch construction strategy designed to curate high-quality batches for Contrastive Learning (CL)<n>Our approach begins by using a pretrained teacher embedding model to rank all examples in the dataset.<n>A community detection algorithm is then applied to this graph to identify clusters of examples that serve as strong negatives for one another.<n>The clusters are then used to construct batches that are rich in in-batch negatives.
• Score: 57.352097333505476
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive learning (CL) is a prevalent technique for training embedding models, which pulls semantically similar examples (positives) closer in the representation space while pushing dissimilar ones (negatives) further apart. A key source of negatives are 'in-batch' examples, i.e., positives from other examples in the batch. Effectiveness of such models is hence strongly influenced by the size and quality of training batches. In this work, we propose 'Breaking the Batch Barrier' (B3), a novel batch construction strategy designed to curate high-quality batches for CL. Our approach begins by using a pretrained teacher embedding model to rank all examples in the dataset, from which a sparse similarity graph is constructed. A community detection algorithm is then applied to this graph to identify clusters of examples that serve as strong negatives for one another. The clusters are then used to construct batches that are rich in in-batch negatives. Empirical results on the MMEB multimodal embedding benchmark (36 tasks) demonstrate that our method sets a new state of the art, outperforming previous best methods by +1.3 and +2.9 points at the 7B and 2B model scales, respectively. Notably, models trained with B3 surpass existing state-of-the-art results even with a batch size as small as 64, which is 4-16x smaller than that required by other methods.

Related papers

• Language Model Preference Evaluation with Multiple Weak Evaluators [78.53743237977677]
  GED (Preference Graph Ensemble and Denoise) is a novel approach that leverages multiple model-based evaluators to construct preference graphs.<n>In particular, our method consists of two primary stages: aggregating evaluations into a unified graph and applying a denoising process.<n>We provide theoretical guarantees for our framework, demonstrating its efficacy in recovering the ground truth preference structure.
  arXiv  Detail & Related papers  (2024-10-14T01:57:25Z)
• Conan-embedding: General Text Embedding with More and Better Negative Samples [30.571206231457932]
  We propose a conan-embedding model, which maximizes the utilization of more and higher-quality negative examples. Our approach effectively enhances the capabilities of embedding models, currently ranking first on the Chinese leaderboard of Massive text embedding benchmark.
  arXiv  Detail & Related papers  (2024-08-28T11:18:06Z)
• MSVQ: Self-Supervised Learning with Multiple Sample Views and Queues [10.327408694770709]
  We propose a new simple framework, namely Multiple Sample Views and Queues (MSVQ) We jointly construct three soft labels on-the-fly by utilizing two complementary and symmetric approaches. Let the student network mimic the similarity relationships between the samples, thus giving the student network a more flexible ability to identify false negative samples in the dataset.
  arXiv  Detail & Related papers  (2023-05-09T12:05:14Z)
• You Only Need End-to-End Training for Long-Tailed Recognition [8.789819609485225]
  Cross-entropy loss tends to produce highly correlated features on imbalanced data. We propose two novel modules, Block-based Relatively Balanced Batch Sampler (B3RS) and Batch Embedded Training (BET) Experimental results on the long-tailed classification benchmarks, CIFAR-LT and ImageNet-LT, demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2021-12-11T11:44:09Z)
• Mixture Proportion Estimation and PU Learning: A Modern Approach [47.34499672878859]
  Given only positive examples and unlabeled examples, we might hope to estimate an accurate positive-versus-negative classifier. classical methods for both problems break down in high-dimensional settings. We propose two simple techniques: Best Bin Estimation (BBE) and Value Ignoring Risk (CVIR)
  arXiv  Detail & Related papers  (2021-11-01T14:42:23Z)
• Neighborhood Contrastive Learning for Novel Class Discovery [79.14767688903028]
  We build a new framework, named Neighborhood Contrastive Learning, to learn discriminative representations that are important to clustering performance. We experimentally demonstrate that these two ingredients significantly contribute to clustering performance and lead our model to outperform state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2021-06-20T17:34:55Z)
• Solving Inefficiency of Self-supervised Representation Learning [87.30876679780532]
  Existing contrastive learning methods suffer from very low learning efficiency. Under-clustering and over-clustering problems are major obstacles to learning efficiency. We propose a novel self-supervised learning framework using a median triplet loss.
  arXiv  Detail & Related papers  (2021-04-18T07:47:10Z)
• Doubly Contrastive Deep Clustering [135.7001508427597]
  We present a novel Doubly Contrastive Deep Clustering (DCDC) framework, which constructs contrastive loss over both sample and class views. Specifically, for the sample view, we set the class distribution of the original sample and its augmented version as positive sample pairs. For the class view, we build the positive and negative pairs from the sample distribution of the class. In this way, two contrastive losses successfully constrain the clustering results of mini-batch samples in both sample and class level.
  arXiv  Detail & Related papers  (2021-03-09T15:15:32Z)
• Few-shot Action Recognition with Prototype-centered Attentive Learning [88.10852114988829]
  Prototype-centered Attentive Learning (PAL) model composed of two novel components. First, a prototype-centered contrastive learning loss is introduced to complement the conventional query-centered learning objective. Second, PAL integrates a attentive hybrid learning mechanism that can minimize the negative impacts of outliers.
  arXiv  Detail & Related papers  (2021-01-20T11:48:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.