From pre-training to downstream performance: Does domain-specific pre-training make sense?

• URL: http://arxiv.org/abs/2605.08819v1
• Date: Sat, 09 May 2026 09:11:02 GMT
• Title: From pre-training to downstream performance: Does domain-specific pre-training make sense?
• Abstract summary: The study underscores the importance of pre-training strategies to enhance the reliability and effectiveness of deep learning models in medical imaging.<n>By addressing these key factors, our research aims to contribute to the development of more accurate and dependable diagnostic tools.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning techniques have revolutionised medical imaging, improving diagnostic accuracy and enabling both more accurate and earlier disease detection. However, the relationship between pre-training strategies and downstream performance in medical imaging models requires further exploration. Here, we systematically compare convolutional neural networks and transformers, examining various pre-training approaches, including supervised and self-supervised learning, as well as different initialisations and data modalities. Models are evaluated on natural images, chest X-rays, chest CT and retina OCT images, considering the effects of matching pre-training data with target modalities. Our findings indicate that only pre-training on data closely matching the target modality significantly improves downstream performance. While self-supervised learning can outperform supervised methods, its effectiveness varies with context. The study underscores the importance of pre-training strategies to enhance the reliability and effectiveness of deep learning models in medical imaging. By addressing these key factors, our research aims to contribute to the development of more accurate and dependable diagnostic tools, ultimately improving patient outcomes in clinical settings.

Related papers

• Entropy-Guided Self-Supervised Learning for Medical Image Classification [0.0]
  This paper introduces a synergistic deep learning framework that leverages the strengths of self-supervised learning and transfer learning for enhanced medical image classification.<n>Our approach employs two distinct ConvNeXt-Tiny models: one pre-trained on a large-scale natural image dataset (ImageNet) and another pre-trained using an entropy-guided Masked Autoencoder (MAE) on the target medical dataset.<n> Rigorous experimental validation across four diverse medical imaging datasets demonstrates the superior performance and robustness of our ensemble approach.
  arXiv  Detail & Related papers  (2026-05-21T04:01:06Z)
• Self-Supervised Learning for Pre-training Capsule Networks: Overcoming Medical Imaging Dataset Challenges [2.9248916859490173]
  This study investigates self-supervised learning methods for pre-training capsule networks in polyp diagnostics for colon cancer.<n>We used the PICCOLO dataset, comprising 3,433 samples, which exemplifies typical challenges in medical datasets.<n>Our findings suggest contrastive learning and in-painting techniques are suitable auxiliary tasks for self-supervised learning in the medical domain.
  arXiv  Detail & Related papers  (2025-02-07T08:32:26Z)
• Transfer or Self-Supervised? Bridging the Performance Gap in Medical Imaging [6.744847405966574]
  This paper compares the performance and robustness of transfer learning and self-supervised learning in the medical field.<n>We tested data with several common issues in medical domains, such as data imbalance, data scarcity, and domain mismatch.<n>We provide recommendations to help users apply transfer learning and self-supervised learning methods in medical areas.
  arXiv  Detail & Related papers  (2024-07-08T04:14:52Z)
• Boosting Few-Shot Learning with Disentangled Self-Supervised Learning and Meta-Learning for Medical Image Classification [8.975676404678374]
  We present a strategy for improving the performance and generalization capabilities of models trained in low-data regimes. The proposed method starts with a pre-training phase, where features learned in a self-supervised learning setting are disentangled to improve the robustness of the representations for downstream tasks. We then introduce a meta-fine-tuning step, leveraging related classes between meta-training and meta-testing phases but varying the level.
  arXiv  Detail & Related papers  (2024-03-26T09:36:20Z)
• A Survey of the Impact of Self-Supervised Pretraining for Diagnostic Tasks with Radiological Images [71.26717896083433]
  Self-supervised pretraining has been observed to be effective at improving feature representations for transfer learning. This review summarizes recent research into its usage in X-ray, computed tomography, magnetic resonance, and ultrasound imaging.
  arXiv  Detail & Related papers  (2023-09-05T19:45:09Z)
• Robust and Efficient Medical Imaging with Self-Supervision [80.62711706785834]
  We present REMEDIS, a unified representation learning strategy to improve robustness and data-efficiency of medical imaging AI. We study a diverse range of medical imaging tasks and simulate three realistic application scenarios using retrospective data.
  arXiv  Detail & Related papers  (2022-05-19T17:34:18Z)
• Performance or Trust? Why Not Both. Deep AUC Maximization with Self-Supervised Learning for COVID-19 Chest X-ray Classifications [72.52228843498193]
  In training deep learning models, a compromise often must be made between performance and trust. In this work, we integrate a new surrogate loss with self-supervised learning for computer-aided screening of COVID-19 patients.
  arXiv  Detail & Related papers  (2021-12-14T21:16:52Z)
• On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy [70.71457102672545]
  We compare the impact of different training procedures for diabetic retinopathy grading. We investigate different aspects such as quantitative performance, statistics of the learned feature representations, interpretability and robustness to image distortions. Our results indicate that models from ImageNet pretraining report a significant increase in performance, generalization and robustness to image distortions.
  arXiv  Detail & Related papers  (2021-06-25T08:32:45Z)
• A Multi-Stage Attentive Transfer Learning Framework for Improving COVID-19 Diagnosis [49.3704402041314]
  We propose a multi-stage attentive transfer learning framework for improving COVID-19 diagnosis. Our proposed framework consists of three stages to train accurate diagnosis models through learning knowledge from multiple source tasks and data of different domains. Importantly, we propose a novel self-supervised learning method to learn multi-scale representations for lung CT images.
  arXiv  Detail & Related papers  (2021-01-14T01:39:19Z)
• Retinopathy of Prematurity Stage Diagnosis Using Object Segmentation and Convolutional Neural Networks [68.96150598294072]
  Retinopathy of Prematurity (ROP) is an eye disorder primarily affecting premature infants with lower weights. It causes proliferation of vessels in the retina and could result in vision loss and, eventually, retinal detachment, leading to blindness. In recent years, there has been a significant effort to automate the diagnosis using deep learning. This paper builds upon the success of previous models and develops a novel architecture, which combines object segmentation and convolutional neural networks (CNN) Our proposed system first trains an object segmentation model to identify the demarcation line at a pixel level and adds the resulting mask as an additional "color" channel in
  arXiv  Detail & Related papers  (2020-04-03T14:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.