Pretext Matters: An Empirical Study of SSL Methods in Medical Imaging

• URL: http://arxiv.org/abs/2603.22649v1
• Date: Mon, 23 Mar 2026 23:53:16 GMT
• Title: Pretext Matters: An Empirical Study of SSL Methods in Medical Imaging
• Abstract summary: Self-supervised learning (SSL) has demonstrated incredible ability to learn robust representations from unlabeled data.<n>The choice of optimal SSL strategy can lead to vastly different performance outcomes in specialized domains.<n>In this work, we empirically investigate how the choice of SSL method impacts the learned representations in medical imaging.
• Score: 3.7670460987350016
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Though self-supervised learning (SSL) has demonstrated incredible ability to learn robust representations from unlabeled data, the choice of optimal SSL strategy can lead to vastly different performance outcomes in specialized domains. Joint embedding architectures (JEAs) and joint embedding predictive architectures (JEPAs) have shown robustness to noise and strong semantic feature learning compared to pixel reconstruction-based SSL methods, leading to widespread adoption in medical imaging. However, no prior work has systematically investigated which SSL objective is better aligned with the spatial organization of clinically relevant signal. In this work, we empirically investigate how the choice of SSL method impacts the learned representations in medical imaging. We select two representative imaging modalities characterized by unique noise profiles: ultrasound and histopathology. When informative signal is spatially localized, as in histopathology, JEAs are more effective due to their view-invariance objective. In contrast, when diagnostically relevant information is globally structured, such as the macroscopic anatomy present in liver ultrasounds, JEPAs are optimal. These differences are especially evident in the clinical relevance of the learned features, as independently validated by board-certified radiologists and pathologists. Together, our results provide a framework for matching SSL objectives to the structural and noise properties of medical imaging modalities.

Related papers

• Task-Aligned Self-Supervised Learning for Medical Image Analysis: A Systematic Review and Practical Design Guidelines [0.0]
  Self-supervised learning has emerged as a promising paradigm for addressing the annotation bottleneck in medical imaging.<n>We present a systematic, task-oriented review of SSL in medical imaging.<n>We examine how different pretext-task formulations influence performance across classification, segmentation, detection, and other tasks.
  arXiv  Detail & Related papers  (2026-05-18T04:47:50Z)
• Multi-View Synergistic Learning with Vision-Language Adaption for Low-Resource Biomedical Image Classification [59.24009931000134]
  Multi-View Synergistic Learning (MVSL) is a unified framework that addresses adaptation paradigms, representation granularity, and disease semantic relationships.<n>MVSL decouples the adaptation of visual and textual encoders to respect their distinct representational characteristics.<n>It further introduces multi-granularity contrastive learning to explicitly model both global image semantics and localized lesion-level evidence.<n>MVSL consistently outperforms state-of-the-art methods in few-shot and zero-shot classification settings.
  arXiv  Detail & Related papers  (2026-04-27T02:41:27Z)
• DiSSECT: Structuring Transfer-Ready Medical Image Representations through Discrete Self-Supervision [9.254163621425727]
  DiSSECT is a framework that integrates multi-scale vector quantization into the SSL pipeline to impose a discrete representational bottleneck.<n>It achieves strong performance on both classification and segmentation tasks, requiring minimal or no fine-tuning.<n>We validate DiSSECT across multiple public medical imaging datasets, demonstrating its robustness and generalizability.
  arXiv  Detail & Related papers  (2025-09-23T07:58:21Z)
• ACE: Anatomically Consistent Embeddings in Composition and Decomposition [5.939793479232325]
  This paper introduces a novel self-supervised learning (SSL) approach called ACE to learn anatomically consistent embedding via composition and decomposition.<n> Experimental results across 6 datasets 2 backbones, evaluated in few-shot learning, fine-tuning, and property analysis, show ACE's superior robustness, transferability, and clinical potential.
  arXiv  Detail & Related papers  (2025-01-17T11:39:47Z)
• OTCXR: Rethinking Self-supervised Alignment using Optimal Transport for Chest X-ray Analysis [6.4136876268620115]
  Self-supervised learning (SSL) has emerged as a promising technique for analyzing medical modalities such as X-rays.<n>We propose OTCXR, a novel SSL framework that leverages optimal transport (OT) to learn dense semantic invariance.<n>We validate OTCXR's efficacy through comprehensive experiments on three publicly available chest X-ray datasets.
  arXiv  Detail & Related papers  (2024-04-18T02:59:48Z)
• Overcoming Dimensional Collapse in Self-supervised Contrastive Learning for Medical Image Segmentation [2.6764957223405657]
  We investigate the application of contrastive learning to the domain of medical image analysis. Our findings reveal that MoCo v2, a state-of-the-art contrastive learning method, encounters dimensional collapse when applied to medical images. To address this, we propose two key contributions: local feature learning and feature decorrelation.
  arXiv  Detail & Related papers  (2024-02-22T15:02:13Z)
• Show from Tell: Audio-Visual Modelling in Clinical Settings [58.88175583465277]
  We consider audio-visual modelling in a clinical setting, providing a solution to learn medical representations without human expert annotation. A simple yet effective multi-modal self-supervised learning framework is proposed for this purpose. The proposed approach is able to localise anatomical regions of interest during ultrasound imaging, with only speech audio as a reference.
  arXiv  Detail & Related papers  (2023-10-25T08:55:48Z)
• K-Space-Aware Cross-Modality Score for Synthesized Neuroimage Quality Assessment [71.27193056354741]
  The problem of how to assess cross-modality medical image synthesis has been largely unexplored. We propose a new metric K-CROSS to spur progress on this challenging problem. K-CROSS uses a pre-trained multi-modality segmentation network to predict the lesion location.
  arXiv  Detail & Related papers  (2023-07-10T01:26:48Z)
• PCRLv2: A Unified Visual Information Preservation Framework for Self-supervised Pre-training in Medical Image Analysis [56.63327669853693]
  We propose to incorporate the task of pixel restoration for explicitly encoding more pixel-level information into high-level semantics. We also address the preservation of scale information, a powerful tool in aiding image understanding. The proposed unified SSL framework surpasses its self-supervised counterparts on various tasks.
  arXiv  Detail & Related papers  (2023-01-02T17:47:27Z)
• Benchmarking Self-Supervised Learning on Diverse Pathology Datasets [10.868779327544688]
  Self-supervised learning has shown to be an effective method for utilizing unlabeled data. We execute the largest-scale study of SSL pre-training on pathology image data. For the first time, we apply SSL to the challenging task of nuclei instance segmentation.
  arXiv  Detail & Related papers  (2022-12-09T06:38:34Z)
• Explaining Clinical Decision Support Systems in Medical Imaging using Cycle-Consistent Activation Maximization [112.2628296775395]
  Clinical decision support using deep neural networks has become a topic of steadily growing interest. clinicians are often hesitant to adopt the technology because its underlying decision-making process is considered to be intransparent and difficult to comprehend. We propose a novel decision explanation scheme based on CycleGAN activation which generates high-quality visualizations of classifier decisions even in smaller data sets.
  arXiv  Detail & Related papers  (2020-10-09T14:39:27Z)
• Uncovering the structure of clinical EEG signals with self-supervised learning [64.4754948595556]
  Supervised learning paradigms are often limited by the amount of labeled data that is available. This phenomenon is particularly problematic in clinically-relevant data, such as electroencephalography (EEG) By extracting information from unlabeled data, it might be possible to reach competitive performance with deep neural networks.
  arXiv  Detail & Related papers  (2020-07-31T14:34:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.