Improving Self-supervised Learning with Hardness-aware Dynamic Curriculum Learning: An Application to Digital Pathology

• URL: http://arxiv.org/abs/2108.07183v1
• Date: Mon, 16 Aug 2021 15:44:48 GMT
• Title: Improving Self-supervised Learning with Hardness-aware Dynamic Curriculum Learning: An Application to Digital Pathology
• Authors: Chetan L Srinidhi, Anne L Martel
• Abstract summary: Self-supervised learning (SSL) has recently shown tremendous potential to learn generic visual representations useful for many image analysis tasks. The existing SSL methods fail to generalize to downstream tasks when the number of labeled training instances is small or if the domain shift between the transfer domains is significant. This paper attempts to improve self-supervised pretrained representations through the lens of curriculum learning.
• Score: 2.2742357407157847
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised learning (SSL) has recently shown tremendous potential to learn generic visual representations useful for many image analysis tasks. Despite their notable success, the existing SSL methods fail to generalize to downstream tasks when the number of labeled training instances is small or if the domain shift between the transfer domains is significant. In this paper, we attempt to improve self-supervised pretrained representations through the lens of curriculum learning by proposing a hardness-aware dynamic curriculum learning (HaDCL) approach. To improve the robustness and generalizability of SSL, we dynamically leverage progressive harder examples via easy-to-hard and hard-to-very-hard samples during mini-batch downstream fine-tuning. We discover that by progressive stage-wise curriculum learning, the pretrained representations are significantly enhanced and adaptable to both in-domain and out-of-domain distribution data. We performed extensive validation on three histology benchmark datasets on both patch-wise and slide-level classification problems. Our curriculum based fine-tuning yields a significant improvement over standard fine-tuning, with a minimum improvement in area-under-the-curve (AUC) score of 1.7% and 2.2% on in-domain and out-of-domain distribution data, respectively. Further, we empirically show that our approach is more generic and adaptable to any SSL methods and does not impose any additional overhead complexity. Besides, we also outline the role of patch-based versus slide-based curriculum learning in histopathology to provide practical insights into the success of curriculum based fine-tuning of SSL methods. Code will be released at https://github.com/srinidhiPY/ICCVCDPATH2021-ID-8

Related papers

• A Closer Look at Benchmarking Self-Supervised Pre-training with Image Classification [51.35500308126506]
  Self-supervised learning (SSL) is a machine learning approach where the data itself provides supervision, eliminating the need for external labels. We study how classification-based evaluation protocols for SSL correlate and how well they predict downstream performance on different dataset types.
  arXiv  Detail & Related papers  (2024-07-16T23:17:36Z)
• Self-supervised visual learning in the low-data regime: a comparative evaluation [40.27083924454058]
  Self-Supervised Learning (SSL) is a robust training methodology for contemporary Deep Neural Networks (DNNs) This work introduces a taxonomy of modern visual SSL methods, accompanied by detailed explanations and insights regarding the main categories of approaches. For domain-specific downstream tasks, in-domain low-data SSL pretraining outperforms the common approach of large-scale pretraining.
  arXiv  Detail & Related papers  (2024-04-26T07:23:14Z)
• SLCA: Slow Learner with Classifier Alignment for Continual Learning on a Pre-trained Model [73.80068155830708]
  We present an extensive analysis for continual learning on a pre-trained model (CLPM) We propose a simple but extremely effective approach named Slow Learner with Alignment (SLCA) Across a variety of scenarios, our proposal provides substantial improvements for CLPM.
  arXiv  Detail & Related papers  (2023-03-09T08:57:01Z)
• Benchmarking Self-Supervised Learning on Diverse Pathology Datasets [10.868779327544688]
  Self-supervised learning has shown to be an effective method for utilizing unlabeled data. We execute the largest-scale study of SSL pre-training on pathology image data. For the first time, we apply SSL to the challenging task of nuclei instance segmentation.
  arXiv  Detail & Related papers  (2022-12-09T06:38:34Z)
• FUSSL: Fuzzy Uncertain Self Supervised Learning [8.31483061185317]
  Self supervised learning (SSL) has become a very successful technique to harness the power of unlabeled data, with no annotation effort. In this paper, for the first time, we recognize the fundamental limits of SSL coming from the use of a single-supervisory signal. We propose a robust and general standard hierarchical learning/training protocol for any SSL baseline.
  arXiv  Detail & Related papers  (2022-10-28T01:06:10Z)
• Dive into Self-Supervised Learning for Medical Image Analysis: Data, Models and Tasks [8.720079280914169]
  Self-supervised learning has achieved remarkable performance in various medical imaging tasks by dint of priors from massive unlabelled data. We focus on exploiting the capacity of SSL in terms of four realistic and significant issues. We provide a large-scale, in-depth and fine-grained study through extensive experiments on predictive, contrastive, generative and multi-SSL algorithms.
  arXiv  Detail & Related papers  (2022-09-25T06:04:11Z)
• Decoupled Adversarial Contrastive Learning for Self-supervised Adversarial Robustness [69.39073806630583]
  Adversarial training (AT) for robust representation learning and self-supervised learning (SSL) for unsupervised representation learning are two active research fields. We propose a two-stage framework termed Decoupled Adversarial Contrastive Learning (DeACL)
  arXiv  Detail & Related papers  (2022-07-22T06:30:44Z)
• DATA: Domain-Aware and Task-Aware Pre-training [94.62676913928831]
  We present DATA, a simple yet effective NAS approach specialized for self-supervised learning (SSL) Our method achieves promising results across a wide range of computation costs on downstream tasks, including image classification, object detection and semantic segmentation.
  arXiv  Detail & Related papers  (2022-03-17T02:38:49Z)
• Self-supervised Learning is More Robust to Dataset Imbalance [65.84339596595383]
  We investigate self-supervised learning under dataset imbalance. Off-the-shelf self-supervised representations are already more robust to class imbalance than supervised representations. We devise a re-weighted regularization technique that consistently improves the SSL representation quality on imbalanced datasets.
  arXiv  Detail & Related papers  (2021-10-11T06:29:56Z)
• A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classification [38.68079253627819]
  Our benchmark consists of two fine-grained classification datasets obtained by sampling classes from the Aves and Fungi taxonomy. We find that recently proposed SSL methods provide significant benefits, and can effectively use out-of-class data to improve performance when deep networks are trained from scratch. Our work suggests that semi-supervised learning with experts on realistic datasets may require different strategies than those currently prevalent in the literature.
  arXiv  Detail & Related papers  (2021-04-01T17:59:41Z)
• On Data-Augmentation and Consistency-Based Semi-Supervised Learning [77.57285768500225]
  Recently proposed consistency-based Semi-Supervised Learning (SSL) methods have advanced the state of the art in several SSL tasks. Despite these advances, the understanding of these methods is still relatively limited.
  arXiv  Detail & Related papers  (2021-01-18T10:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.