Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging

• URL: http://arxiv.org/abs/2608.00586v1
• Date: Sat, 01 Aug 2026 10:53:26 GMT
• Title: Representation Transfer of Foundation Models for Ultra-Widefield Retinal Imaging
• Abstract summary: We evaluate how different pretraining strategies influence the transferability of learned representations to weakly supervised ophthalmic imaging tasks.<n>We compare Vision Transformer encoders pretrained with supervised, Masked Autoencoder (MAE) and self-distillation objectives.<n>A contemporary DINOv3 model pretrained at a larger scale achieved the strongest overall performance.
• Score: 4.175300836877834
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the widespread adoption of foundation models as feature extractors for medical imaging, relatively little is understood about how different pretraining strategies influence the transferability of learned representations to weakly supervised ophthalmic imaging tasks. We investigate this question in ultra-widefield (UWF) retinal imaging by evaluating foundation model representations within a patch-based multiple instance learning (MIL) framework for disease classification on UWF images. We compare Vision Transformer encoders pretrained with supervised, Masked Autoencoder (MAE), and self-distillation objectives, while keeping the downstream aggregation architecture unchanged. Within a controlled comparison of ViT-B encoders pretrained on ImageNet-1k, the choice of pretraining objective substantially influenced frozen representation transfer, with supervised and self-distillation-based models outperforming MAE. A contemporary DINOv3 model pretrained at a larger scale achieved the strongest overall performance, with a quadratic weighted kappa of 0.863 for five-class diabetic retinopathy grading, comparable with DINOv1. Attention analysis further revealed distinct patch-aggregation behaviours associated with the different pretrained representations, while partial fine-tuning substantially reduced the performance gap for MAE. These findings suggest that pretraining strategy influences both representation transferability and the subsequent aggregation of patch-level evidence within MIL, resulting in differences in downstream classification performance.

Related papers

• Entropy-Guided Self-Supervised Learning for Medical Image Classification [0.0]
  This paper introduces a synergistic deep learning framework that leverages the strengths of self-supervised learning and transfer learning for enhanced medical image classification.<n>Our approach employs two distinct ConvNeXt-Tiny models: one pre-trained on a large-scale natural image dataset (ImageNet) and another pre-trained using an entropy-guided Masked Autoencoder (MAE) on the target medical dataset.<n> Rigorous experimental validation across four diverse medical imaging datasets demonstrates the superior performance and robustness of our ensemble approach.
  arXiv  Detail & Related papers  (2026-05-21T04:01:06Z)
• UniSegDiff: Boosting Unified Lesion Segmentation via a Staged Diffusion Model [53.34835793648352]
  We propose UniSegDiff, a novel diffusion model framework for lesion segmentation.<n>UniSegDiff addresses lesion segmentation in a unified manner across multiple modalities and organs.<n> Comprehensive experimental results demonstrate that UniSegDiff significantly outperforms previous state-of-the-art (SOTA) approaches.
  arXiv  Detail & Related papers  (2025-07-24T12:33:10Z)
• Self-supervised learning improves robustness of deep learning lung tumor segmentation to CT imaging differences [7.332652485849634]
  Self-supervised learning (SSL) is an approach to extract useful feature representations from unlabeled data. We compare robustness of wild versus self-pretrained transformer (ViT) and hierarchical shifted window (Swin) models to computed tomography (CT) imaging differences. Wild-pretrained networks were more robust to analyzed CT imaging differences for lung tumor segmentation than self-pretrained methods.
  arXiv  Detail & Related papers  (2024-05-14T14:35:21Z)
• Adapting Visual-Language Models for Generalizable Anomaly Detection in Medical Images [68.42215385041114]
  This paper introduces a novel lightweight multi-level adaptation and comparison framework to repurpose the CLIP model for medical anomaly detection. Our approach integrates multiple residual adapters into the pre-trained visual encoder, enabling a stepwise enhancement of visual features across different levels. Our experiments on medical anomaly detection benchmarks demonstrate that our method significantly surpasses current state-of-the-art models.
  arXiv  Detail & Related papers  (2024-03-19T09:28:19Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• DiffMIC: Dual-Guidance Diffusion Network for Medical Image Classification [32.67098520984195]
  We propose the first diffusion-based model (named DiffMIC) to address general medical image classification. Our experimental results demonstrate that DiffMIC outperforms state-of-the-art methods by a significant margin.
  arXiv  Detail & Related papers  (2023-03-19T09:15:45Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• GraVIS: Grouping Augmented Views from Independent Sources for Dermatology Analysis [52.04899592688968]
  We propose GraVIS, which is specifically optimized for learning self-supervised features from dermatology images. GraVIS significantly outperforms its transfer learning and self-supervised learning counterparts in both lesion segmentation and disease classification tasks.
  arXiv  Detail & Related papers  (2023-01-11T11:38:37Z)
• A Closer Look at Self-Supervised Lightweight Vision Transformers [44.44888945683147]
  Self-supervised learning on large-scale Vision Transformers (ViTs) as pre-training methods has achieved promising downstream performance. We benchmark several self-supervised pre-training methods on image classification tasks and some downstream dense prediction tasks. Even vanilla lightweight ViTs show comparable performance to previous SOTA networks with delicate architecture design.
  arXiv  Detail & Related papers  (2022-05-28T14:14:57Z)
• How Transferable Are Self-supervised Features in Medical Image Classification Tasks? [0.7734726150561086]
  Transfer learning has become a standard practice to mitigate the lack of labeled data in medical classification tasks. Self-supervised pretrained models yield richer embeddings than their supervised counterpart. Dynamic Visual Meta-Embedding (DVME) is an end-to-end transfer learning approach that fuses pretrained embeddings from multiple models.
  arXiv  Detail & Related papers  (2021-08-23T10:39:31Z)
• On the Robustness of Pretraining and Self-Supervision for a Deep Learning-based Analysis of Diabetic Retinopathy [70.71457102672545]
  We compare the impact of different training procedures for diabetic retinopathy grading. We investigate different aspects such as quantitative performance, statistics of the learned feature representations, interpretability and robustness to image distortions. Our results indicate that models from ImageNet pretraining report a significant increase in performance, generalization and robustness to image distortions.
  arXiv  Detail & Related papers  (2021-06-25T08:32:45Z)
• About Explicit Variance Minimization: Training Neural Networks for Medical Imaging With Limited Data Annotations [2.3204178451683264]
  Variance Aware Training (VAT) method exploits this property by introducing the variance error into the model loss function. We validate VAT on three medical imaging datasets from diverse domains and various learning objectives.
  arXiv  Detail & Related papers  (2021-05-28T21:34:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.