Self-supervised Pre-training Helps Retinal Disease Progression Modelling Most When Data Is Scarce

• URL: http://arxiv.org/abs/2609.12834v1
• Date: Fri, 11 Sep 2026 13:28:36 GMT
• Title: Self-supervised Pre-training Helps Retinal Disease Progression Modelling Most When Data Is Scarce
• Abstract summary: Modelling how a disease progresses requires longitudinal imaging cohorts, which are scarce and small.<n>Self-supervised pre-training on such data offers a way to bridge this gap.<n>We study this for age-related macular degeneration (AMD)<n>We compare in-house self-supervised encoders against a general-purpose (DINOv2) and a domain-specific (RETFound) foundation model.
• Score: 5.183784643227601
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Modelling how a disease progresses over time requires longitudinal imaging cohorts, which are scarce and small, whereas cross-sectional data -- one image per participant -- is abundant. Self-supervised pre-training on such data offers a way to bridge this gap, but it is unclear which strategy best supports progression modelling, or how that answer depends on the amount of labelled longitudinal data. We study this for age-related macular degeneration (AMD), pre-training encoders on the large cross-sectional NAKO cohort and predicting time to late AMD on the longitudinal AREDS dataset. We compare in-house self-supervised encoders against a general-purpose (DINOv2) and a domain-specific (RETFound) foundation model, across contrastive, masked-autoencoding, and self-distillation objectives, under frozen and fine-tuned protocols, and across labelled training sets from 100 to 32,250 examples. Which model performs best depends on how the encoder is used. When the encoder is frozen and labels are few -- the regime typical of longitudinal cohorts -- pre-trained representations reach clinically reasonable discrimination from a few hundred labelled samples, while models trained from scratch do not; this advantage fades under fine-tuning. Transfer is governed by the self-supervision objective rather than corpus scale or domain match, so that an encoder pre-trained on a modest cross-sectional cohort matches or exceeds a far larger in-domain foundation model. Together, these results offer a practical recipe for building progression models where longitudinal data is scarce: a frozen self-supervised encoder with a lightweight survival head.

Related papers

• MAE-Based Self-Supervised Pretraining for Data-Efficient Medical Image Segmentation Using nnFormer [0.5437050212139087]
  Transformer architectures, including nnFormer, have demonstrated promising results in volumetric medical image segmentation.<n>These models need large quantities of labeled training data and are also likely to overfit and become training unstable.<n>This is a serious practical problem because it is not only time-consuming but also expensive to obtain medical images that are annotated by experts.<n>We have solved these drawbacks by advancing the efficiency of the nnFormer with a self-supervised pretraining framework.
  arXiv  Detail & Related papers  (2026-04-22T11:29:08Z)
• The Mean is the Mirage: Entropy-Adaptive Model Merging under Heterogeneous Domain Shifts in Medical Imaging [3.597779662054083]
  Model merging under unseen test-time distribution shifts often renders naive strategies, such as mean averaging unreliable.<n>We introduce an entropy-adaptive, fully online model-merging method that yields a batch-specific merged model via only forward passes.<n>We extensively evaluate our method with state-of-the-art baselines using two backbones across nine medical and natural-domain generalization image classification datasets.
  arXiv  Detail & Related papers  (2026-02-24T21:06:19Z)
• CoDA: From Text-to-Image Diffusion Models to Training-Free Dataset Distillation [71.52209438343928]
  Core Distribution Alignment (CoDA) is a framework that enables effective Distillation (DD) using only an off-the-shelf text-to-image model.<n>Our key idea is to first identify the "intrinsic core distribution" of the target dataset using a robust density-based discovery mechanism.<n>By doing so, CoDA effectively bridges the gap between general-purpose generative priors and target semantics.
  arXiv  Detail & Related papers  (2025-12-03T14:45:57Z)
• Unlabeled Data vs. Pre-trained Knowledge: Rethinking SSL in the Era of Large Models [24.291082472792905]
  Semi-supervised learning (SSL) alleviates the cost of data labeling process by exploiting unlabeled data.<n> exploiting pre-trained models becomes a promising way to address the label scarcity in the downstream tasks.<n>This raises a natural yet critical question: When labeled data is limited, should we rely on unlabeled data or pre-trained models?
  arXiv  Detail & Related papers  (2025-05-19T16:29:20Z)
• Domain Adaptive Diabetic Retinopathy Grading with Model Absence and Flowing Data [45.75724873443564]
  Domain shift poses a significant challenge in clinical applications, e.g., Diabetic Retinopathy grading.<n>We propose a novel approach, Generative Unadversarial ExampleS (GUES), which enables adaptation from a data-centric perspective.
  arXiv  Detail & Related papers  (2024-12-02T07:14:25Z)
• Towards Theoretical Understandings of Self-Consuming Generative Models [56.84592466204185]
  This paper tackles the emerging challenge of training generative models within a self-consuming loop. We construct a theoretical framework to rigorously evaluate how this training procedure impacts the data distributions learned by future models. We present results for kernel density estimation, delivering nuanced insights such as the impact of mixed data training on error propagation.
  arXiv  Detail & Related papers  (2024-02-19T02:08:09Z)
• Refine, Discriminate and Align: Stealing Encoders via Sample-Wise Prototypes and Multi-Relational Extraction [57.16121098944589]
  RDA is a pioneering approach designed to address two primary deficiencies prevalent in previous endeavors aiming at stealing pre-trained encoders. It is accomplished via a sample-wise prototype, which consolidates the target encoder's representations for a given sample's various perspectives. For more potent efficacy, we develop a multi-relational extraction loss that trains the surrogate encoder to Discriminate mismatched embedding-prototype pairs.
  arXiv  Detail & Related papers  (2023-12-01T15:03:29Z)
• Complexity Matters: Rethinking the Latent Space for Generative Modeling [65.64763873078114]
  In generative modeling, numerous successful approaches leverage a low-dimensional latent space, e.g., Stable Diffusion. In this study, we aim to shed light on this under-explored topic by rethinking the latent space from the perspective of model complexity.
  arXiv  Detail & Related papers  (2023-07-17T07:12:29Z)
• Self-Distillation for Further Pre-training of Transformers [83.84227016847096]
  We propose self-distillation as a regularization for a further pre-training stage. We empirically validate the efficacy of self-distillation on a variety of benchmark datasets for image and text classification tasks.
  arXiv  Detail & Related papers  (2022-09-30T02:25:12Z)
• Improving self-supervised pretraining models for epileptic seizure detection from EEG data [0.23624125155742057]
  This paper presents various self-supervision strategies to enhance the performance of a time-series based Diffusion convolution neural network (DCRNN) model. The learned weights in the self-supervision pretraining phase can be transferred to the supervised training phase to boost the model's prediction capability.
  arXiv  Detail & Related papers  (2022-06-28T17:15:49Z)
• SANSformers: Self-Supervised Forecasting in Electronic Health Records with Attention-Free Models [48.07469930813923]
  This work aims to forecast the demand for healthcare services, by predicting the number of patient visits to healthcare facilities. We introduce SANSformer, an attention-free sequential model designed with specific inductive biases to cater for the unique characteristics of EHR data. Our results illuminate the promising potential of tailored attention-free models and self-supervised pretraining in refining healthcare utilization predictions across various patient demographics.
  arXiv  Detail & Related papers  (2021-08-31T08:23:56Z)
• Contrastive Model Inversion for Data-Free Knowledge Distillation [60.08025054715192]
  We propose Contrastive Model Inversion, where the data diversity is explicitly modeled as an optimizable objective. Our main observation is that, under the constraint of the same amount of data, higher data diversity usually indicates stronger instance discrimination. Experiments on CIFAR-10, CIFAR-100, and Tiny-ImageNet demonstrate that CMI achieves significantly superior performance when the generated data are used for knowledge distillation.
  arXiv  Detail & Related papers  (2021-05-18T15:13:00Z)
• Self-Training with Improved Regularization for Sample-Efficient Chest X-Ray Classification [80.00316465793702]
  We present a deep learning framework that enables robust modeling in challenging scenarios. Our results show that using 85% lesser labeled data, we can build predictive models that match the performance of classifiers trained in a large-scale data setting.
  arXiv  Detail & Related papers  (2020-05-03T02:36:00Z)
• Synthetic vascular structure generation for unsupervised pre-training in CTA segmentation tasks [0.0]
  We train a U-net architecture at a vessel segmentation task that can be used to provide insights when treating stroke patients. We create a computational model that generates synthetic vascular structures which can be blended into unlabeled CT scans of the head.
  arXiv  Detail & Related papers  (2020-01-02T23:21:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.