Hard Negative Mixing for Contrastive Learning

• URL: http://arxiv.org/abs/2010.01028v2
• Date: Fri, 4 Dec 2020 11:46:48 GMT
• Title: Hard Negative Mixing for Contrastive Learning
• Authors: Yannis Kalantidis, Mert Bulent Sariyildiz, Noe Pion, Philippe Weinzaepfel, Diane Larlus
• Abstract summary: We argue that an important aspect of contrastive learning, i.e., the effect of hard negatives, has so far been neglected. We propose hard negative mixing strategies at the feature level, that can be computed on-the-fly with a minimal computational overhead.
• Score: 29.91220669060252
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Contrastive learning has become a key component of self-supervised learning approaches for computer vision. By learning to embed two augmented versions of the same image close to each other and to push the embeddings of different images apart, one can train highly transferable visual representations. As revealed by recent studies, heavy data augmentation and large sets of negatives are both crucial in learning such representations. At the same time, data mixing strategies either at the image or the feature level improve both supervised and semi-supervised learning by synthesizing novel examples, forcing networks to learn more robust features. In this paper, we argue that an important aspect of contrastive learning, i.e., the effect of hard negatives, has so far been neglected. To get more meaningful negative samples, current top contrastive self-supervised learning approaches either substantially increase the batch sizes, or keep very large memory banks; increasing the memory size, however, leads to diminishing returns in terms of performance. We therefore start by delving deeper into a top-performing framework and show evidence that harder negatives are needed to facilitate better and faster learning. Based on these observations, and motivated by the success of data mixing, we propose hard negative mixing strategies at the feature level, that can be computed on-the-fly with a minimal computational overhead. We exhaustively ablate our approach on linear classification, object detection and instance segmentation and show that employing our hard negative mixing procedure improves the quality of visual representations learned by a state-of-the-art self-supervised learning method.

Related papers

• Contrastive Learning for Object Detection [0.0]
  We look to improve supervised contrastive learning by ranking classes based on their similarity. We observe the impact of human bias (in the form of ranking) on the learned representations.
  arXiv  Detail & Related papers  (2022-08-12T02:02:23Z)
• Weakly Supervised Contrastive Learning [68.47096022526927]
  We introduce a weakly supervised contrastive learning framework (WCL) to tackle this issue. WCL achieves 65% and 72% ImageNet Top-1 Accuracy using ResNet50, which is even higher than SimCLRv2 with ResNet101.
  arXiv  Detail & Related papers  (2021-10-10T12:03:52Z)
• Semi-Supervising Learning, Transfer Learning, and Knowledge Distillation with SimCLR [2.578242050187029]
  Recent breakthroughs in the field of semi-supervised learning have achieved results that match state-of-the-art traditional supervised learning methods. SimCLR is the current state-of-the-art semi-supervised learning framework for computer vision.
  arXiv  Detail & Related papers  (2021-08-02T01:37:39Z)
• Visual Adversarial Imitation Learning using Variational Models [60.69745540036375]
  Reward function specification remains a major impediment for learning behaviors through deep reinforcement learning. Visual demonstrations of desired behaviors often presents an easier and more natural way to teach agents. We develop a variational model-based adversarial imitation learning algorithm.
  arXiv  Detail & Related papers  (2021-07-16T00:15:18Z)
• Incremental False Negative Detection for Contrastive Learning [95.68120675114878]
  We introduce a novel incremental false negative detection for self-supervised contrastive learning. During contrastive learning, we discuss two strategies to explicitly remove the detected false negatives. Our proposed method outperforms other self-supervised contrastive learning frameworks on multiple benchmarks within a limited compute.
  arXiv  Detail & Related papers  (2021-06-07T15:29:14Z)
• Solving Inefficiency of Self-supervised Representation Learning [87.30876679780532]
  Existing contrastive learning methods suffer from very low learning efficiency. Under-clustering and over-clustering problems are major obstacles to learning efficiency. We propose a novel self-supervised learning framework using a median triplet loss.
  arXiv  Detail & Related papers  (2021-04-18T07:47:10Z)
• Contrastive Learning based Hybrid Networks for Long-Tailed Image Classification [31.647639786095993]
  We propose a novel hybrid network structure composed of a supervised contrastive loss to learn image representations and a cross-entropy loss to learn classifiers. Experiments on three long-tailed classification datasets demonstrate the advantage of the proposed contrastive learning based hybrid networks in long-tailed classification.
  arXiv  Detail & Related papers  (2021-03-26T05:22:36Z)
• Delving into Inter-Image Invariance for Unsupervised Visual Representations [108.33534231219464]
  We present a study to better understand the role of inter-image invariance learning. Online labels converge faster than offline labels. Semi-hard negative samples are more reliable and unbiased than hard negative samples.
  arXiv  Detail & Related papers  (2020-08-26T17:44:23Z)
• On Mutual Information in Contrastive Learning for Visual Representations [19.136685699971864]
  unsupervised, "contrastive" learning algorithms in vision have been shown to learn representations that perform remarkably well on transfer tasks. We show that this family of algorithms maximizes a lower bound on the mutual information between two or more "views" of an image. We find that the choice of negative samples and views are critical to the success of these algorithms.
  arXiv  Detail & Related papers  (2020-05-27T04:21:53Z)
• Un-Mix: Rethinking Image Mixtures for Unsupervised Visual Representation Learning [108.999497144296]
  Recently advanced unsupervised learning approaches use the siamese-like framework to compare two "views" from the same image for learning representations. This work aims to involve the distance concept on label space in the unsupervised learning and let the model be aware of the soft degree of similarity between positive or negative pairs. Despite its conceptual simplicity, we show empirically that with the solution -- Unsupervised image mixtures (Un-Mix), we can learn subtler, more robust and generalized representations from the transformed input and corresponding new label space.
  arXiv  Detail & Related papers  (2020-03-11T17:59:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.