Self-supervision drives representational convergence in medical foundation models more than clinical supervision

• URL: http://arxiv.org/abs/2607.20274v1
• Date: Wed, 22 Jul 2026 15:25:05 GMT
• Title: Self-supervision drives representational convergence in medical foundation models more than clinical supervision
• Abstract summary: We present a controlled dissection across 18 image and 7 text encoders, all open-weight and run locally, spanning 7M to 27B parameters and five imaging modalities.<n>We train encoders that vary only the objective under fixed data, architecture, and scale, and reproduce the effect in a synthetic model.<n> Convergence is modest but above a random floor, driven by the self-supervised objective, not clinical supervision.
• Score: 2.8356496885865017
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Medical image encoders from different groups are increasingly treated as interchangeable, on the assumption that scale and clinical supervision concentrate their representations onto a shared structure. Whether this convergence is real, what produces it, and whether it is clinically usable are untested, and the similarity measures behind such claims are fragile. We present a controlled dissection across 18 image and 7 text encoders, all open-weight and run locally, spanning 7M to 27B parameters and five imaging modalities, including 650,982 chest radiographs from six datasets. To isolate cause, we train encoders that vary only the objective under fixed data, architecture, and scale, and reproduce the effect in a synthetic model. Convergence is modest but above a random floor, driven by the self-supervised objective, not clinical supervision: matched self-supervised encoders aligned most (40.4% on chest radiography), with label-supervised (21.1%) and image-text (3.3%) far lower, and did not grow with size (Spearman 0.302, p=0.223) or capability. It is within-modality, does not reach clinical language, and does not reproduce how radiologists judge case similarity. Yet a linear classifier transfers across encoders and to five held-out hospitals, retaining about 85% of within-encoder performance. Convergence in medical imaging is therefore set by the pretraining objective, not inherited from scale or clinical supervision. Interoperability is accordingly something to design for through that objective, and to validate where the shared geometry is weakest, across patient subgroups and against clinical judgment.

Related papers

• Anatomical Grounding and Leakage-Aware Multimodal Contrastive Learning for Alzheimer's Disease Classification from Structural MRI [0.0]
  Deep networks trained on structural MRI for Alzheimer's disease often reach reasonable accuracy while attending to anatomically irrelevant regions.<n>We use FastSurfer segmentations as an anatomical reference.<n>We adapt a CLIP-style image and organize ADNIMERGE variables along a label-leakage spectrum.
  arXiv  Detail & Related papers  (2026-09-14T17:13:12Z)
• Do Medical Vision Models Reason About Anatomy? Probing the Spatial Inductive Biases of Learned Visual Representations [0.0]
  We construct SPAR-Bench, eight probes over multi-organ abdominal CT.<n>We apply them to five architectural configurations and three medical foundation models, frozen and finetuned.<n>Our results suggest these encoders carry a map of where organs usually lie, and little of the machinery for comparing structures within a particular patient.
  arXiv  Detail & Related papers  (2026-08-28T08:59:27Z)
• Rethinking Artificial Intelligence in Medical Imaging: Assumptions, Reality, and Reframing [29.25526289599026]
  We argue that a decade of intense research has not translated into proportionate bedside impact.<n>This Perspective identifies six interconnected dimensions of this misalignment.<n>We propose a vision of agentic, physician-aligned AI that extends, rather than replaces, clinical judgment.
  arXiv  Detail & Related papers  (2026-07-29T19:55:49Z)
• A Vision-language Framework for Comparative Reasoning in Radiology [68.52471827773482]
  We formulate radiological comparison as an entity-aware cross-image reasoning problem.<n>We construct a large-scale comparative imaging resource derived from routine image-report pairs.<n>We develop MedReCo, an entity-aware visual encoder for controllable retrieval of clinically analogous cases.
  arXiv  Detail & Related papers  (2026-06-04T17:12:47Z)
• Methodology for Creating a Clinically Verified Dermoscopic Image Dataset [0.0]
  This study presents a methodology for constructing a clinically verified dataset of dermatoscopic images for medical informatics research.<n>International collections were primarily created under conditions that differ substantially from routine Russian outpatient practice and mobile dermatoscopy.<n>Using this methodology, a dataset of 1,026 unique dermatoscopic images from 443 patients was collected between June 2025 and May 2026.<n>The dataset includes nine nosological categories; all 39 malignant lesions were histologically verified.
  arXiv  Detail & Related papers  (2026-05-24T16:56:47Z)
• TriALS: Triphasic-Aided Liver Lesion Segmentation Benchmark in Non-Contrast CT [49.93685997940356]
  We describe the TriALS challenge for automated liver lesion segmentation under contrast-limited conditions.<n>The top-performing method achieved a mean venous-phase Dice of 0.754, consistent with human-level performance, yet dropped to 0.57 on NCCT.<n>Algorithm performance was most strongly predicted by training data scale and pre-training strategy.
  arXiv  Detail & Related papers  (2026-05-15T19:23:35Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• Performance of a Deep Learning-Based Segmentation Model for Pancreatic Tumors on Public Endoscopic Ultrasound Datasets [2.925528117330222]
  Pancreatic cancer is one of the most aggressive cancers, with poor survival rates.<n>This study evaluates a Vision Transformer-based deep learning segmentation model for pancreatic tumors.
  arXiv  Detail & Related papers  (2026-01-09T16:48:50Z)
• Click, Predict, Trust: Clinician-in-the-Loop AI Segmentation for Lung Cancer CT-Based Prognosis within the Knowledge-to-Action Framework [0.7340285482623061]
  Lung cancer remains the leading cause of cancer mortality, with CT imaging central to screening, prognosis, and treatment.<n>Deep learning (DL) offers automation but faces barriers to clinical adoption.<n>This study develops a clinician-in-the-loop DL pipeline to enhance, prognostic accuracy, and clinical trust.
  arXiv  Detail & Related papers  (2025-10-19T23:02:43Z)
• TotalSegmentator MRI: Robust Sequence-independent Segmentation of Multiple Anatomic Structures in MRI [59.86827659781022]
  A nnU-Net model (TotalSegmentator) was trained on MRI and segment 80atomic structures.<n>Dice scores were calculated between the predicted segmentations and expert reference standard segmentations to evaluate model performance.<n>Open-source, easy-to-use model allows for automatic, robust segmentation of 80 structures.
  arXiv  Detail & Related papers  (2024-05-29T20:15:54Z)
• Rethinking Semi-Supervised Medical Image Segmentation: A Variance-Reduction Perspective [51.70661197256033]
  We propose ARCO, a semi-supervised contrastive learning framework with stratified group theory for medical image segmentation. We first propose building ARCO through the concept of variance-reduced estimation and show that certain variance-reduction techniques are particularly beneficial in pixel/voxel-level segmentation tasks. We experimentally validate our approaches on eight benchmarks, i.e., five 2D/3D medical and three semantic segmentation datasets, with different label settings.
  arXiv  Detail & Related papers  (2023-02-03T13:50:25Z)
• Significantly improving zero-shot X-ray pathology classification via fine-tuning pre-trained image-text encoders [50.689585476660554]
  We propose a new fine-tuning strategy that includes positive-pair loss relaxation and random sentence sampling. Our approach consistently improves overall zero-shot pathology classification across four chest X-ray datasets and three pre-trained models.
  arXiv  Detail & Related papers  (2022-12-14T06:04:18Z)
• Malignancy Prediction and Lesion Identification from Clinical Dermatological Images [65.1629311281062]
  We consider machine-learning-based malignancy prediction and lesion identification from clinical dermatological images. We first identify all lesions present in the image regardless of sub-type or likelihood of malignancy, then it estimates their likelihood of malignancy, and through aggregation, it also generates an image-level likelihood of malignancy.
  arXiv  Detail & Related papers  (2021-04-02T20:52:05Z)
• Chest x-ray automated triage: a semiologic approach designed for clinical implementation, exploiting different types of labels through a combination of four Deep Learning architectures [83.48996461770017]
  This work presents a Deep Learning method based on the late fusion of different convolutional architectures. We built four training datasets combining images from public chest x-ray datasets and our institutional archive. We trained four different Deep Learning architectures and combined their outputs with a late fusion strategy, obtaining a unified tool.
  arXiv  Detail & Related papers  (2020-12-23T14:38:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.