Rethinking Positive Pairs in Contrastive Learning

• URL: http://arxiv.org/abs/2410.18200v1
• Date: Wed, 23 Oct 2024 18:07:18 GMT
• Title: Rethinking Positive Pairs in Contrastive Learning
• Authors: Jiantao Wu, Shentong Mo, Zhenhua Feng, Sara Atito, Josef Kitler, Muhammad Awais,
• Abstract summary: We present Hydra, a universal contrastive learning framework for visual representations that extends conventional contrastive learning to accommodate arbitrary pairs. Our approach is validated using IN1K, where 1K diverse classes compose 500,500 pairs, most of them being distinct. Our work highlights the value of learning common features of arbitrary pairs and potentially broadens the applicability of contrastive learning techniques on the sample pairs with weak relationships.
• Score: 19.149235307036324
• License:
• Abstract: Contrastive learning, a prominent approach to representation learning, traditionally assumes positive pairs are closely related samples (the same image or class) and negative pairs are distinct samples. We challenge this assumption by proposing to learn from arbitrary pairs, allowing any pair of samples to be positive within our framework.The primary challenge of the proposed approach lies in applying contrastive learning to disparate pairs which are semantically distant. Motivated by the discovery that SimCLR can separate given arbitrary pairs (e.g., garter snake and table lamp) in a subspace, we propose a feature filter in the condition of class pairs that creates the requisite subspaces by gate vectors selectively activating or deactivating dimensions. This filter can be optimized through gradient descent within a conventional contrastive learning mechanism. We present Hydra, a universal contrastive learning framework for visual representations that extends conventional contrastive learning to accommodate arbitrary pairs. Our approach is validated using IN1K, where 1K diverse classes compose 500,500 pairs, most of them being distinct. Surprisingly, Hydra achieves superior performance in this challenging setting. Additional benefits include the prevention of dimensional collapse and the discovery of class relationships. Our work highlights the value of learning common features of arbitrary pairs and potentially broadens the applicability of contrastive learning techniques on the sample pairs with weak relationships.

Related papers

• CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective [48.99488315273868]
  We present a contrastive knowledge distillation approach, which can be formulated as a sample-wise alignment problem with intra- and inter-sample constraints. Our method minimizes logit differences within the same sample by considering their numerical values. We conduct comprehensive experiments on three datasets including CIFAR-100, ImageNet-1K, and MS COCO.
  arXiv  Detail & Related papers  (2024-04-22T11:52:40Z)
• Prototypical Contrastive Learning through Alignment and Uniformity for Recommendation [6.790779112538357]
  We present underlinePrototypical contrastive learning through underlineAlignment and underlineUniformity for recommendation. Specifically, we first propose prototypes as a latent space to ensure consistency across different augmentations from the origin graph. The absence of explicit negatives means that directly optimizing the consistency loss between instance and prototype could easily result in dimensional collapse issues.
  arXiv  Detail & Related papers  (2024-02-03T08:19:26Z)
• Learning by Sorting: Self-supervised Learning with Group Ordering Constraints [75.89238437237445]
  This paper proposes a new variation of the contrastive learning objective, Group Ordering Constraints (GroCo) It exploits the idea of sorting the distances of positive and negative pairs and computing the respective loss based on how many positive pairs have a larger distance than the negative pairs, and thus are not ordered correctly. We evaluate the proposed formulation on various self-supervised learning benchmarks and show that it not only leads to improved results compared to vanilla contrastive learning but also shows competitive performance to comparable methods in linear probing and outperforms current methods in k-NN performance.
  arXiv  Detail & Related papers  (2023-01-05T11:17:55Z)
• Non-contrastive representation learning for intervals from well logs [58.70164460091879]
  The representation learning problem in the oil & gas industry aims to construct a model that provides a representation based on logging data for a well interval. One of the possible approaches is self-supervised learning (SSL) We are the first to introduce non-contrastive SSL for well-logging data.
  arXiv  Detail & Related papers  (2022-09-28T13:27:10Z)
• Adaptive Soft Contrastive Learning [19.45520684918576]
  This paper proposes an adaptive method that introduces soft inter-sample relations, namely Adaptive Soft Contrastive Learning (ASCL) As an effective and concise plug-in module for existing self-supervised learning frameworks, ASCL achieves the best performance on several benchmarks.
  arXiv  Detail & Related papers  (2022-07-22T16:01:07Z)
• Contrasting quadratic assignments for set-based representation learning [5.142415132534397]
  standard approach to contrastive learning is to maximize the agreement between different views of the data. In this work, we note that the approach of considering individual pairs cannot account for both intra-set and inter-set similarities. We propose to go beyond contrasting individual pairs of objects by focusing on contrasting objects as sets.
  arXiv  Detail & Related papers  (2022-05-31T14:14:36Z)
• Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap [64.60460828425502]
  We propose a new guarantee on the downstream performance of contrastive learning. Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations. We propose an unsupervised model selection metric ARC that aligns well with downstream accuracy.
  arXiv  Detail & Related papers  (2022-03-25T05:36:26Z)
• A Low Rank Promoting Prior for Unsupervised Contrastive Learning [108.91406719395417]
  We construct a novel probabilistic graphical model that effectively incorporates the low rank promoting prior into the framework of contrastive learning. Our hypothesis explicitly requires that all the samples belonging to the same instance class lie on the same subspace with small dimension. Empirical evidences show that the proposed algorithm clearly surpasses the state-of-the-art approaches on multiple benchmarks.
  arXiv  Detail & Related papers  (2021-08-05T15:58:25Z)
• Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss [72.62029620566925]
  Recent works in self-supervised learning have advanced the state-of-the-art by relying on the contrastive learning paradigm. Our work analyzes contrastive learning without assuming conditional independence of positive pairs. We propose a loss that performs spectral decomposition on the population augmentation graph and can be succinctly written as a contrastive learning objective.
  arXiv  Detail & Related papers  (2021-06-08T07:41:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.