MoCo-Transfer: Investigating out-of-distribution contrastive learning for limited-data domains

• URL: http://arxiv.org/abs/2311.09401v1
• Date: Wed, 15 Nov 2023 21:56:47 GMT
• Title: MoCo-Transfer: Investigating out-of-distribution contrastive learning for limited-data domains
• Authors: Yuwen Chen, Helen Zhou, Zachary C. Lipton
• Abstract summary: We analyze the benefit of transferring self-supervised contrastive representations from moment contrast (MoCo) pretraining to settings with limited data. We find that depending on quantity of labeled and unlabeled data, contrastive pretraining on larger out-of-distribution datasets can perform nearly as well or better than MoCo pretraining in-domain.
• Score: 52.612507614610244
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Medical imaging data is often siloed within hospitals, limiting the amount of data available for specialized model development. With limited in-domain data, one might hope to leverage larger datasets from related domains. In this paper, we analyze the benefit of transferring self-supervised contrastive representations from moment contrast (MoCo) pretraining on out-of-distribution data to settings with limited data. We consider two X-ray datasets which image different parts of the body, and compare transferring from each other to transferring from ImageNet. We find that depending on quantity of labeled and unlabeled data, contrastive pretraining on larger out-of-distribution datasets can perform nearly as well or better than MoCo pretraining in-domain, and pretraining on related domains leads to higher performance than if one were to use the ImageNet pretrained weights. Finally, we provide a preliminary way of quantifying similarity between datasets.

Related papers

• Precision at Scale: Domain-Specific Datasets On-Demand [3.5900418884504095]
  Precision at Scale (PaS) is a novel method for the autonomous creation of domain-specific datasets on-demand. PaS pipeline enables leveraging state-of-the-art foundational and generative models to create a collection of images belonging to any given domain. We prove that automatically generated domain-specific datasets lead to better pretraining than large-scale supervised datasets such as ImageNet-1k and ImageNet-21k.
  arXiv  Detail & Related papers  (2024-07-03T19:17:42Z)
• Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery [78.43828998065071]
  Recent advances in unsupervised learning have demonstrated the ability of large vision models to achieve promising results on downstream tasks. Such pre-training techniques have also been explored recently in the remote sensing domain due to the availability of large amount of unlabelled data. In this paper, we re-visit transformers pre-training and leverage multi-scale information that is effectively utilized with multiple modalities.
  arXiv  Detail & Related papers  (2024-03-08T16:18:04Z)
• Is in-domain data beneficial in transfer learning for landmarks detection in x-ray images? [1.5348047288817481]
  We study whether the usage of small-scale in-domain x-ray image datasets may provide any improvement for landmark detection over models pre-trained on large natural image datasets only. Our results show that using in-domain source datasets brings marginal or no benefit with respect to an ImageNet out-of-domain pre-training. Our findings can provide an indication for the development of robust landmark detection systems in medical images when no large annotated dataset is available.
  arXiv  Detail & Related papers  (2024-03-03T10:35:00Z)
• On the Connection between Pre-training Data Diversity and Fine-tuning Robustness [66.30369048726145]
  We find that the primary factor influencing downstream effective robustness is data quantity. We demonstrate our findings on pre-training distributions drawn from various natural and synthetic data sources.
  arXiv  Detail & Related papers  (2023-07-24T05:36:19Z)
• Why does my medical AI look at pictures of birds? Exploring the efficacy of transfer learning across domain boundaries [3.259272366439713]
  Pretraining on data from the domain of the downstream task should almost always be preferred instead of ImageNet-pretrained models. We leverage RadNet-12M, a dataset containing more than 12 million computed tomography (CT) image slices. Our experiments cover intra- and cross-domain transfer scenarios, varying data scales, finetuning vs. linear evaluation, and feature space analysis.
  arXiv  Detail & Related papers  (2023-06-30T11:15:26Z)
• Self-Supervised In-Domain Representation Learning for Remote Sensing Image Scene Classification [1.0152838128195465]
  Transferring the ImageNet pre-trained weights to the various remote sensing tasks has produced acceptable results. Recent research has demonstrated that self-supervised learning methods capture visual features that are more discriminative and transferable. We are motivated by these facts to pre-train the in-domain representations of remote sensing imagery using contrastive self-supervised learning.
  arXiv  Detail & Related papers  (2023-02-03T15:03:07Z)
• Self-Supervised Pre-Training for Transformer-Based Person Re-Identification [54.55281692768765]
  Transformer-based supervised pre-training achieves great performance in person re-identification (ReID) Due to the domain gap between ImageNet and ReID datasets, it usually needs a larger pre-training dataset to boost the performance. This work aims to mitigate the gap between the pre-training and ReID datasets from the perspective of data and model structure.
  arXiv  Detail & Related papers  (2021-11-23T18:59:08Z)
• Multi-dataset Pretraining: A Unified Model for Semantic Segmentation [97.61605021985062]
  We propose a unified framework, termed as Multi-Dataset Pretraining, to take full advantage of the fragmented annotations of different datasets. This is achieved by first pretraining the network via the proposed pixel-to-prototype contrastive loss over multiple datasets. In order to better model the relationship among images and classes from different datasets, we extend the pixel level embeddings via cross dataset mixing.
  arXiv  Detail & Related papers  (2021-06-08T06:13:11Z)
• Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training [67.71228426496013]
  We show that using target domain data during pre-training leads to large performance improvements across a variety of setups. We find that pre-training on multiple domains improves performance generalization on domains not seen during training.
  arXiv  Detail & Related papers  (2021-04-02T12:53:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.