Self-supervised visual learning in the low-data regime: a comparative evaluation

• URL: http://arxiv.org/abs/2404.17202v2
• Date: Thu, 26 Dec 2024 14:17:40 GMT
• Title: Self-supervised visual learning in the low-data regime: a comparative evaluation
• Authors: Sotirios Konstantakos, Jorgen Cani, Ioannis Mademlis, Despina Ioanna Chalkiadaki, Yuki M. Asano, Efstratios Gavves, Georgios Th. Papadopoulos,
• Abstract summary: Self-Supervised Learning (SSL) is a valuable and robust training methodology for contemporary Deep Neural Networks (DNNs) It allows efficient representation learning from massive amounts of unlabeled training data. It is not always feasible to collect and/or to utilize very large pretraining datasets.
• Score: 38.34785825702943
• License:
• Abstract: Self-Supervised Learning (SSL) is a valuable and robust training methodology for contemporary Deep Neural Networks (DNNs), enabling unsupervised pretraining on a 'pretext task' that does not require ground-truth labels/annotation. This allows efficient representation learning from massive amounts of unlabeled training data, which in turn leads to increased accuracy in a 'downstream task' by exploiting supervised transfer learning. Despite the relatively straightforward conceptualization and applicability of SSL, it is not always feasible to collect and/or to utilize very large pretraining datasets, especially when it comes to real-world application settings. In particular, in cases of specialized and domain-specific application scenarios, it may not be achievable or practical to assemble a relevant image pretraining dataset in the order of millions of instances or it could be computationally infeasible to pretrain at this scale, e.g., due to unavailability of sufficient computational resources that SSL methods typically require to produce improved visual analysis results. This situation motivates an investigation on the effectiveness of common SSL pretext tasks, when the pretraining dataset is of relatively limited/constrained size. This work briefly introduces the main families of modern visual SSL methods and, subsequently, conducts a thorough comparative experimental evaluation in the low-data regime, targeting to identify: a) what is learnt via low-data SSL pretraining, and b) how do different SSL categories behave in such training scenarios. Interestingly, for domain-specific downstream tasks, in-domain low-data SSL pretraining outperforms the common approach of large-scale pretraining on general datasets.

Related papers

• A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification [51.35500308126506]
  Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels. We study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types.
  arXiv  Detail & Related papers  (2024-07-16T23:17:36Z)
• Self-supervised learning for skin cancer diagnosis with limited training data [0.196629787330046]
  Self-supervised learning (SSL) is an alternative to the standard supervised pre-training on ImageNet for scenarios with limited training data. We consider textitfurther SSL pre-training on task-specific datasets, where our implementation is motivated by supervised transfer learning. We find minimal further SSL pre-training on task-specific data can be as effective as large-scale SSL pre-training on ImageNet for medical image classification tasks with limited labelled data.
  arXiv  Detail & Related papers  (2024-01-01T08:11:38Z)
• Progressive Feature Adjustment for Semi-supervised Learning from Pretrained Models [39.42802115580677]
  Semi-supervised learning (SSL) can leverage both labeled and unlabeled data to build a predictive model. Recent literature suggests that naively applying state-of-the-art SSL with a pretrained model fails to unleash the full potential of training data. We propose to use pseudo-labels from the unlabelled data to update the feature extractor that is less sensitive to incorrect labels.
  arXiv  Detail & Related papers  (2023-09-09T01:57:14Z)
• In-Domain Self-Supervised Learning Improves Remote Sensing Image Scene Classification [5.323049242720532]
  Self-supervised learning has emerged as a promising approach for remote sensing image classification. We present a study of different self-supervised pre-training strategies and evaluate their effect across 14 downstream datasets.
  arXiv  Detail & Related papers  (2023-07-04T10:57:52Z)
• Task-Customized Self-Supervised Pre-training with Scalable Dynamic Routing [76.78772372631623]
  A common practice for self-supervised pre-training is to use as much data as possible. For a specific downstream task, however, involving irrelevant data in pre-training may degenerate the downstream performance. It is burdensome and infeasible to use different downstream-task-customized datasets in pre-training for different tasks.
  arXiv  Detail & Related papers  (2022-05-26T10:49:43Z)
• DATA: Domain-Aware and Task-Aware Pre-training [94.62676913928831]
  We present DATA, a simple yet effective NAS approach specialized for self-supervised learning (SSL) Our method achieves promising results across a wide range of computation costs on downstream tasks, including image classification, object detection and semantic segmentation.
  arXiv  Detail & Related papers  (2022-03-17T02:38:49Z)
• Self-supervised Learning is More Robust to Dataset Imbalance [65.84339596595383]
  We investigate self-supervised learning under dataset imbalance. Off-the-shelf self-supervised representations are already more robust to class imbalance than supervised representations. We devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets.
  arXiv  Detail & Related papers  (2021-10-11T06:29:56Z)
• Trash to Treasure: Harvesting OOD Data with Cross-Modal Matching for Open-Set Semi-Supervised Learning [101.28281124670647]
  Open-set semi-supervised learning (open-set SSL) investigates a challenging but practical scenario where out-of-distribution (OOD) samples are contained in the unlabeled data. We propose a novel training mechanism that could effectively exploit the presence of OOD data for enhanced feature learning. Our approach substantially lifts the performance on open-set SSL and outperforms the state-of-the-art by a large margin.
  arXiv  Detail & Related papers  (2021-08-12T09:14:44Z)
• Transfer Learning or Self-supervised Learning? A Tale of Two Pretraining Paradigms [36.04356511882304]
  Self-supervised learning (SSL) has demonstrated promising results on a wide range of applications. There has not been a clear understanding on what properties of data and tasks render one approach outperforms the other.
  arXiv  Detail & Related papers  (2020-06-19T05:21:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.