Center-wise Local Image Mixture For Contrastive Representation Learning

• URL: http://arxiv.org/abs/2011.02697v3
• Date: Mon, 18 Oct 2021 02:15:36 GMT
• Title: Center-wise Local Image Mixture For Contrastive Representation Learning
• Authors: Hao Li, Xiaopeng Zhang, Hongkai Xiong
• Abstract summary: Contrastive learning based on instance discrimination trains model to discriminate different transformations of the anchor sample from other samples. This paper proposes a new kind of contrastive learning method, named CLIM, which uses positives from other samples in the dataset. We reach 75.5% top-1 accuracy with linear evaluation over ResNet-50, and 59.3% top-1 accuracy when fine-tuned with only 1% labels.
• Score: 37.806687971373954
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Contrastive learning based on instance discrimination trains model to discriminate different transformations of the anchor sample from other samples, which does not consider the semantic similarity among samples. This paper proposes a new kind of contrastive learning method, named CLIM, which uses positives from other samples in the dataset. This is achieved by searching local similar samples of the anchor, and selecting samples that are closer to the corresponding cluster center, which we denote as center-wise local image selection. The selected samples are instantiated via an data mixture strategy, which performs as a smoothing regularization. As a result, CLIM encourages both local similarity and global aggregation in a robust way, which we find is beneficial for feature representation. Besides, we introduce \emph{multi-resolution} augmentation, which enables the representation to be scale invariant. We reach 75.5% top-1 accuracy with linear evaluation over ResNet-50, and 59.3% top-1 accuracy when fine-tuned with only 1% labels.

Related papers

• CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective [48.99488315273868]
  We present a contrastive knowledge distillation approach, which can be formulated as a sample-wise alignment problem with intra- and inter-sample constraints. Our method minimizes logit differences within the same sample by considering their numerical values. We conduct comprehensive experiments on three datasets including CIFAR-100, ImageNet-1K, and MS COCO.
  arXiv  Detail & Related papers  (2024-04-22T11:52:40Z)
• Pairwise Similarity Distribution Clustering for Noisy Label Learning [0.0]
  Noisy label learning aims to train deep neural networks using a large amount of samples with noisy labels. We propose a simple yet effective sample selection algorithm to divide the training samples into one clean set and another noisy set. Experimental results on various benchmark datasets, such as CIFAR-10, CIFAR-100 and Clothing1M, demonstrate significant improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-04-02T11:30:22Z)
• Saliency Grafting: Innocuous Attribution-Guided Mixup with Calibrated Label Mixing [104.630875328668]
  Mixup scheme suggests mixing a pair of samples to create an augmented training sample. We present a novel, yet simple Mixup-variant that captures the best of both worlds.
  arXiv  Detail & Related papers  (2021-12-16T11:27:48Z)
• With a Little Help from My Friends: Nearest-Neighbor Contrastive Learning of Visual Representations [87.72779294717267]
  Using the nearest-neighbor as positive in contrastive losses improves performance significantly on ImageNet classification. We demonstrate empirically that our method is less reliant on complex data augmentations.
  arXiv  Detail & Related papers  (2021-04-29T17:56:08Z)
• Transform consistency for learning with noisy labels [9.029861710944704]
  We propose a method to identify clean samples only using one single network. Clean samples prefer to reach consistent predictions for the original images and the transformed images. In order to mitigate the negative influence of noisy labels, we design a classification loss by using the off-line hard labels and on-line soft labels.
  arXiv  Detail & Related papers  (2021-03-25T14:33:13Z)
• Doubly Contrastive Deep Clustering [135.7001508427597]
  We present a novel Doubly Contrastive Deep Clustering (DCDC) framework, which constructs contrastive loss over both sample and class views. Specifically, for the sample view, we set the class distribution of the original sample and its augmented version as positive sample pairs. For the class view, we build the positive and negative pairs from the sample distribution of the class. In this way, two contrastive losses successfully constrain the clustering results of mini-batch samples in both sample and class level.
  arXiv  Detail & Related papers  (2021-03-09T15:15:32Z)
• Permuted AdaIN: Reducing the Bias Towards Global Statistics in Image Classification [97.81205777897043]
  Recent work has shown that convolutional neural network classifiers overly rely on texture at the expense of shape cues. We make a similar but different distinction between shape and local image cues, on the one hand, and global image statistics, on the other. Our method, called Permuted Adaptive Instance Normalization (pAdaIN), reduces the representation of global statistics in the hidden layers of image classifiers.
  arXiv  Detail & Related papers  (2020-10-09T16:38:38Z)
• Unsupervised Representation Learning by InvariancePropagation [34.53866045440319]
  In this paper, we propose Invariance propagation to focus on learning representations invariant to category-level variations. With a ResNet-50 as the backbone, our method achieves 71.3% top-1 accuracy on ImageNet linear classification and 78.2% top-5 accuracy fine-tuning on only 1% labels. We also achieve state-of-the-art performance on other downstream tasks, including linear classification on Places205 and Pascal VOC, and transfer learning on small scale datasets.
  arXiv  Detail & Related papers  (2020-10-07T13:00:33Z)
• Conditional Negative Sampling for Contrastive Learning of Visual Representations [19.136685699971864]
  We show that choosing difficult negatives, or those more similar to the current instance, can yield stronger representations. We introduce a family of mutual information estimators that sample negatives conditionally -- in a "ring" around each positive. We prove that these estimators lower-bound mutual information, with higher bias but lower variance than NCE.
  arXiv  Detail & Related papers  (2020-10-05T14:17:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.