Bag of Instances Aggregation Boosts Self-supervised Learning

• URL: http://arxiv.org/abs/2107.01691v1
• Date: Sun, 4 Jul 2021 17:33:59 GMT
• Title: Bag of Instances Aggregation Boosts Self-supervised Learning
• Authors: Haohang Xu and Jiemin Fang and Xiaopeng Zhang and Lingxi Xie and Xinggang Wang and Wenrui Dai and Hongkai Xiong and Qi Tian
• Abstract summary: We propose a simple but effective distillation strategy for unsupervised learning. Our method, termed as BINGO, targets at transferring the relationship learned by the teacher to the student. BINGO achieves new state-of-the-art performance on small scale models.
• Score: 122.61914701794296
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent advances in self-supervised learning have experienced remarkable progress, especially for contrastive learning based methods, which regard each image as well as its augmentations as an individual class and try to distinguish them from all other images. However, due to the large quantity of exemplars, this kind of pretext task intrinsically suffers from slow convergence and is hard for optimization. This is especially true for small scale models, which we find the performance drops dramatically comparing with its supervised counterpart. In this paper, we propose a simple but effective distillation strategy for unsupervised learning. The highlight is that the relationship among similar samples counts and can be seamlessly transferred to the student to boost the performance. Our method, termed as BINGO, which is short for \textbf{B}ag of \textbf{I}nsta\textbf{N}ces a\textbf{G}gregati\textbf{O}n, targets at transferring the relationship learned by the teacher to the student. Here bag of instances indicates a set of similar samples constructed by the teacher and are grouped within a bag, and the goal of distillation is to aggregate compact representations over the student with respect to instances in a bag. Notably, BINGO achieves new state-of-the-art performance on small scale models, \emph{i.e.}, 65.5% and 68.9% top-1 accuracies with linear evaluation on ImageNet, using ResNet-18 and ResNet-34 as backbone, respectively, surpassing baselines (52.5% and 57.4% top-1 accuracies) by a significant margin. The code will be available at \url{https://github.com/haohang96/bingo}.

Related papers

• $\mathbb{X}$-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs [62.565573316667276]
  We develop an objective that encodes how a sample relates to others. We train vision models based on similarities in class or text caption descriptions. Our objective appears to work particularly well in lower-data regimes, with gains over CLIP of $16.8%$ on ImageNet and $18.1%$ on ImageNet Real.
  arXiv  Detail & Related papers  (2024-07-25T15:38:16Z)
• Intra-task Mutual Attention based Vision Transformer for Few-Shot Learning [12.5354658533836]
  Humans possess remarkable ability to accurately classify new, unseen images after being exposed to only a few examples. For artificial neural network models, determining the most relevant features for distinguishing between two images with limited samples presents a challenge. We propose an intra-task mutual attention method for few-shot learning, that involves splitting the support and query samples into patches.
  arXiv  Detail & Related papers  (2024-05-06T02:02:57Z)
• CoNe: Contrast Your Neighbours for Supervised Image Classification [62.12074282211957]
  Contrast Your Neighbours (CoNe) is a learning framework for supervised image classification. CoNe employs the features of its similar neighbors as anchors to generate more adaptive and refined targets. Our CoNe achieves 80.8% Top-1 accuracy on ImageNet with ResNet-50, which surpasses the recent Timm training recipe.
  arXiv  Detail & Related papers  (2023-08-21T14:49:37Z)
• MoBYv2AL: Self-supervised Active Learning for Image Classification [57.4372176671293]
  We present MoBYv2AL, a novel self-supervised active learning framework for image classification. Our contribution lies in lifting MoBY, one of the most successful self-supervised learning algorithms, to the AL pipeline. We achieve state-of-the-art results when compared to recent AL methods.
  arXiv  Detail & Related papers  (2023-01-04T10:52:02Z)
• Weakly Supervised Contrastive Learning [68.47096022526927]
  We introduce a weakly supervised contrastive learning framework (WCL) to tackle this issue. WCL achieves 65% and 72% ImageNet Top-1 Accuracy using ResNet50, which is even higher than SimCLRv2 with ResNet101.
  arXiv  Detail & Related papers  (2021-10-10T12:03:52Z)
• CompRess: Self-Supervised Learning by Compressing Representations [14.739041141948032]
  We develop a model compression method to compress an already learned, deep self-supervised model (teacher) to a smaller one (student) We train the student model so that it mimics the relative similarity between the data points in the teacher's embedding space. This is the first time a self-supervised AlexNet has outperformed supervised one on ImageNet classification.
  arXiv  Detail & Related papers  (2020-10-28T02:49:18Z)
• Memory-Augmented Relation Network for Few-Shot Learning [114.47866281436829]
  In this work, we investigate a new metric-learning method, Memory-Augmented Relation Network (MRN) In MRN, we choose the samples that are visually similar from the working context, and perform weighted information propagation to attentively aggregate helpful information from chosen ones to enhance its representation. We empirically demonstrate that MRN yields significant improvement over its ancestor and achieves competitive or even better performance when compared with other few-shot learning approaches.
  arXiv  Detail & Related papers  (2020-05-09T10:09:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.