MoPro: Webly Supervised Learning with Momentum Prototypes

• URL: http://arxiv.org/abs/2009.07995v1
• Date: Thu, 17 Sep 2020 00:59:59 GMT
• Title: MoPro: Webly Supervised Learning with Momentum Prototypes
• Authors: Junnan Li, Caiming Xiong, Steven C.H. Hoi
• Abstract summary: We propose a webly-supervised representation learning method that does not suffer from the annotation unscalability of supervised learning. MoPro achieves state-of-the-art performance on WebVision, a weakly-labeled noisy dataset.
• Score: 140.76848620407168
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a webly-supervised representation learning method that does not suffer from the annotation unscalability of supervised learning, nor the computation unscalability of self-supervised learning. Most existing works on webly-supervised representation learning adopt a vanilla supervised learning method without accounting for the prevalent noise in the training data, whereas most prior methods in learning with label noise are less effective for real-world large-scale noisy data. We propose momentum prototypes (MoPro), a simple contrastive learning method that achieves online label noise correction, out-of-distribution sample removal, and representation learning. MoPro achieves state-of-the-art performance on WebVision, a weakly-labeled noisy dataset. MoPro also shows superior performance when the pretrained model is transferred to down-stream image classification and detection tasks. It outperforms the ImageNet supervised pretrained model by +10.5 on 1-shot classification on VOC, and outperforms the best self-supervised pretrained model by +17.3 when finetuned on 1\% of ImageNet labeled samples. Furthermore, MoPro is more robust to distribution shifts. Code and pretrained models are available at https://github.com/salesforce/MoPro.

Related papers

• Self-Supervised Learning in Deep Networks: A Pathway to Robust Few-Shot Classification [0.0]
  We first pre-train the model with self-supervision to enable it to learn common feature expressions on a large amount of unlabeled data. Then fine-tune it on the few-shot dataset Mini-ImageNet to improve the model's accuracy and generalization ability under limited data.
  arXiv  Detail & Related papers  (2024-11-19T01:01:56Z)
• Pre-Trained Vision-Language Models as Partial Annotators [40.89255396643592]
  Pre-trained vision-language models learn massive data to model unified representations of images and natural languages. In this paper, we investigate a novel "pre-trained annotating - weakly-supervised learning" paradigm for pre-trained model application and experiment on image classification tasks.
  arXiv  Detail & Related papers  (2024-05-23T17:17:27Z)
• EfficientTrain++: Generalized Curriculum Learning for Efficient Visual Backbone Training [79.96741042766524]
  We reformulate the training curriculum as a soft-selection function. We show that exposing the contents of natural images can be readily achieved by the intensity of data augmentation. The resulting method, EfficientTrain++, is simple, general, yet surprisingly effective.
  arXiv  Detail & Related papers  (2024-05-14T17:00:43Z)
• A Simple and Efficient Baseline for Data Attribution on Images [107.12337511216228]
  Current state-of-the-art approaches require a large ensemble of as many as 300,000 models to accurately attribute model predictions. In this work, we focus on a minimalist baseline, utilizing the feature space of a backbone pretrained via self-supervised learning to perform data attribution. Our method is model-agnostic and scales easily to large datasets.
  arXiv  Detail & Related papers  (2023-11-03T17:29:46Z)
• On-the-Fly Guidance Training for Medical Image Registration [14.309599960641242]
  This study introduces a novel On-the-Fly Guidance (OFG) training framework for enhancing existing learning-based image registration models. Our method proposes a supervised fashion for training registration models, without the need for any labeled data. Our method is tested across several benchmark datasets and leading models, it significantly enhanced performance.
  arXiv  Detail & Related papers  (2023-08-29T11:12:53Z)
• The effectiveness of MAE pre-pretraining for billion-scale pretraining [65.98338857597935]
  We introduce an additional pre-pretraining stage that is simple and uses the self-supervised MAE technique to initialize the model. We measure the effectiveness of pre-pretraining on 10 different visual recognition tasks spanning image classification, video recognition, object detection, low-shot classification and zero-shot recognition.
  arXiv  Detail & Related papers  (2023-03-23T17:56:12Z)
• EfficientTrain: Exploring Generalized Curriculum Learning for Training Visual Backbones [80.662250618795]
  This paper presents a new curriculum learning approach for the efficient training of visual backbones (e.g., vision Transformers) As an off-the-shelf method, it reduces the wall-time training cost of a wide variety of popular models by >1.5x on ImageNet-1K/22K without sacrificing accuracy.
  arXiv  Detail & Related papers  (2022-11-17T17:38:55Z)
• Masked Unsupervised Self-training for Zero-shot Image Classification [98.23094305347709]
  Masked Unsupervised Self-Training (MUST) is a new approach which leverages two different and complimentary sources of supervision: pseudo-labels and raw images. MUST improves upon CLIP by a large margin and narrows the performance gap between unsupervised and supervised classification.
  arXiv  Detail & Related papers  (2022-06-07T02:03:06Z)
• Beyond Self-Supervision: A Simple Yet Effective Network Distillation Alternative to Improve Backbones [40.33419553042038]
  We propose to improve existing baseline networks via knowledge distillation from off-the-shelf pre-trained big powerful models. Our solution performs distillation by only driving prediction of the student model consistent with that of the teacher model. We empirically find that such simple distillation settings perform extremely effective, for example, the top-1 accuracy on ImageNet-1k validation set of MobileNetV3-large and ResNet50-D can be significantly improved.
  arXiv  Detail & Related papers  (2021-03-10T09:32:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.