MixSiam: A Mixture-based Approach to Self-supervised Representation Learning

• URL: http://arxiv.org/abs/2111.02679v1
• Date: Thu, 4 Nov 2021 08:12:47 GMT
• Title: MixSiam: A Mixture-based Approach to Self-supervised Representation Learning
• Authors: Xiaoyang Guo, Tianhao Zhao, Yutian Lin, Bo Du
• Abstract summary: Recently contrastive learning has shown significant progress in learning visual representations from unlabeled data. We propose MixSiam, a mixture-based approach upon the traditional siamese network.
• Score: 33.52892899982186
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently contrastive learning has shown significant progress in learning visual representations from unlabeled data. The core idea is training the backbone to be invariant to different augmentations of an instance. While most methods only maximize the feature similarity between two augmented data, we further generate more challenging training samples and force the model to keep predicting discriminative representation on these hard samples. In this paper, we propose MixSiam, a mixture-based approach upon the traditional siamese network. On the one hand, we input two augmented images of an instance to the backbone and obtain the discriminative representation by performing an element-wise maximum of two features. On the other hand, we take the mixture of these augmented images as input, and expect the model prediction to be close to the discriminative representation. In this way, the model could access more variant data samples of an instance and keep predicting invariant discriminative representations for them. Thus the learned model is more robust compared to previous contrastive learning methods. Extensive experiments on large-scale datasets show that MixSiam steadily improves the baseline and achieves competitive results with state-of-the-art methods. Our code will be released soon.

Related papers

• Exploring Data Augmentations on Self-/Semi-/Fully- Supervised Pre-trained Models [24.376036129920948]
  We investigate how data augmentation affects performance of vision pre-trained models. We apply 4 types of data augmentations termed with Random Erasing, CutOut, CutMix and MixUp. We report their performance on vision tasks such as image classification, object detection, instance segmentation, and semantic segmentation.
  arXiv  Detail & Related papers  (2023-10-28T23:46:31Z)
• DiffAug: Enhance Unsupervised Contrastive Learning with Domain-Knowledge-Free Diffusion-based Data Augmentation [48.25619775814776]
  This paper proposes DiffAug, a novel unsupervised contrastive learning technique with diffusion mode-based positive data generation. DiffAug consists of a semantic encoder and a conditional diffusion model; the conditional diffusion model generates new positive samples conditioned on the semantic encoding. Experimental evaluations show that DiffAug outperforms hand-designed and SOTA model-based augmentation methods on DNA sequence, visual, and bio-feature datasets.
  arXiv  Detail & Related papers  (2023-09-10T13:28:46Z)
• Self-Evolution Learning for Mixup: Enhance Data Augmentation on Few-Shot Text Classification Tasks [75.42002070547267]
  We propose a self evolution learning (SE) based mixup approach for data augmentation in text classification. We introduce a novel instance specific label smoothing approach, which linearly interpolates the model's output and one hot labels of the original samples to generate new soft for label mixing up.
  arXiv  Detail & Related papers  (2023-05-22T23:43:23Z)
• The Trade-off between Universality and Label Efficiency of Representations from Contrastive Learning [32.15608637930748]
  We show that there exists a trade-off between the two desiderata so that one may not be able to achieve both simultaneously. We provide analysis using a theoretical data model and show that, while more diverse pre-training data result in more diverse features for different tasks, it puts less emphasis on task-specific features.
  arXiv  Detail & Related papers  (2023-02-28T22:14:33Z)
• Self-similarity Driven Scale-invariant Learning for Weakly Supervised Person Search [66.95134080902717]
  We propose a novel one-step framework, named Self-similarity driven Scale-invariant Learning (SSL) We introduce a Multi-scale Exemplar Branch to guide the network in concentrating on the foreground and learning scale-invariant features. Experiments on PRW and CUHK-SYSU databases demonstrate the effectiveness of our method.
  arXiv  Detail & Related papers  (2023-02-25T04:48:11Z)
• DoubleMix: Simple Interpolation-Based Data Augmentation for Text Classification [56.817386699291305]
  This paper proposes a simple yet effective data augmentation approach termed DoubleMix. DoubleMix first generates several perturbed samples for each training data. It then uses the perturbed data and original data to carry out a two-step in the hidden space of neural models.
  arXiv  Detail & Related papers  (2022-09-12T15:01:04Z)
• Self-supervised Pre-training with Hard Examples Improves Visual Representations [110.23337264762512]
  Self-supervised pre-training (SSP) employs random image transformations to generate training data for visual representation learning. We first present a modeling framework that unifies existing SSP methods as learning to predict pseudo-labels. Then, we propose new data augmentation methods of generating training examples whose pseudo-labels are harder to predict than those generated via random image transformations.
  arXiv  Detail & Related papers  (2020-12-25T02:44:22Z)
• Demystifying Contrastive Self-Supervised Learning: Invariances, Augmentations and Dataset Biases [34.02639091680309]
  Recent gains in performance come from training instance classification models, treating each image and it's augmented versions as samples of a single class. We demonstrate that approaches like MOCO and PIRL learn occlusion-invariant representations. Second, we demonstrate that these approaches obtain further gains from access to a clean object-centric training dataset like Imagenet.
  arXiv  Detail & Related papers  (2020-07-28T00:11:31Z)
• CSI: Novelty Detection via Contrastive Learning on Distributionally Shifted Instances [77.28192419848901]
  We propose a simple, yet effective method named contrasting shifted instances (CSI) In addition to contrasting a given sample with other instances as in conventional contrastive learning methods, our training scheme contrasts the sample with distributionally-shifted augmentations of itself. Our experiments demonstrate the superiority of our method under various novelty detection scenarios.
  arXiv  Detail & Related papers  (2020-07-16T08:32:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.