Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging

• URL: http://arxiv.org/abs/2605.14654v1
• Date: Thu, 14 May 2026 10:10:34 GMT
• Title: Beyond Instance-Level Self-Supervision in 3D Multi-Modal Medical Imaging
• Abstract summary: Self-supervised pre-training methods in medical imaging typically treat each individual as an isolated instance.<n>We propose leveraging this cross-instance topological consistency as a supervisory signal.<n>We validate our approach across 7 downstream multi-modal tasks, achieving average improvements of 1.1% and 5.94% in segmentation and classification tasks.
• Score: 29.556385532527752
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Self-supervised pre-training methods in medical imaging typically treat each individual as an isolated instance, learning representations through augmentation-based objectives or masked reconstruction. They often do not adequately capitalize on a key characteristic of physiological features: anatomical structures maintain consistent spatial relationships across individuals (instances), such as the thalamus being medial to the basal ganglia, regardless of variations in brain size, shape, or pathology. We propose leveraging this cross-instance topological consistency as a supervisory signal. The challenge arises from the inherent variability in medical imaging, which can differ significantly across instances and modalities. To tackle this, we focus on two alignment regimes. (i) Intra-instance: with pixel-level correspondences available, a cross-modal triplet objective explicitly preserves local neighborhood topology. (ii) Inter-instance: without such supervision, we derive pseudo-correspondences to control partial neighborhood alignment and prevent topology collapse across modalities. We validate our approach across 7 downstream multi-modal tasks, achieving average improvements of 1.1% and 5.94% in segmentation and classification tasks, respectively, and demonstrating significantly better robustness when modalities are missing at test time.

Related papers

• Benchmarking Intra-Patient 3D Deformable Multimodal Image Registration [4.37027575905308]
  We present a benchmark of intra-patient 3D multimodal deformable registration methods across three datasets.<n>We evaluate classical optimization-based approaches and modern learning-based methods, including recent deep learning and foundation models.<n>Results show high variability across datasets, with learning-based methods demonstrating superior performance on large synthetic benchmarks.
  arXiv  Detail & Related papers  (2026-09-14T14:47:57Z)
• Dementia Etiology Diagnosis via Collaborative Meta Knowledge Enhancement [81.27196033392559]
  We propose a Collaborative Meta Knowledge Enhancement (COME) framework for dementia etiology diagnosis.<n>COME injects multi-center acquisition semantics, source identifiers, and modality indicators as heterogeneity-aware embeddings into a unified Transformer architecture for scale-up training.<n>Our method achieves state-of-the-art in-domain performance with a mean macro-averaged AUC of 85.62% and a 4.29-point gain over the strongest baseline.
  arXiv  Detail & Related papers  (2026-07-24T04:26:47Z)
• Topology-Driven Transferability Estimation for 3D Medical Vision Foundation Models [24.834339924064505]
  We propose a voxel-wise framework for dense prediction tasks in 3D medical imaging.<n>Our framework estimates transferability directly from the alignment between the sparse 1-skeleton geometric graph of dense features and semantic labels.<n>Our framework achieves state-of-the-art transferability estimation with an average weighted Kendall (outperforming by 0.36) while accelerating evaluation by 56 times.
  arXiv  Detail & Related papers  (2026-07-05T09:45:18Z)
• PRIME: Prototype-Driven Multimodal Pretraining for Cancer Prognosis with Missing Modalities [86.63247982275396]
  PRIME is a missing-aware multimodal self-supervised pretraining framework.<n>It learns robust and transferable representations from partially observed cohorts.<n>We evaluate PRIME on The Cancer Genome Atlas with label-free pretraining on 32 cancer types.
  arXiv  Detail & Related papers  (2026-04-05T21:14:27Z)
• Multicentric thrombus segmentation using an attention-based recurrent network with gradual modality dropout [0.0]
  We introduce a methodology that couples an attention-based recurrent segmentation network (UpAttLLSTM)<n>UpAttLLSTM aggregates context across slices via recurrent units (2.5D) and uses attention gates to fuse complementary cues across available sequences.<n>On a monocentric cohort, our approach detects thrombi in >90% of cases with a Dice score of 0.65.<n>In a multi-center setting with missing modalities, it achieves-80% detection with a Dice score around 0.35.
  arXiv  Detail & Related papers  (2026-04-01T12:25:00Z)
• PF-DAformer: Proximal Femur Segmentation via Domain Adaptive Transformer for Dual-Center QCT [8.358409792893278]
  We develop a domain-adaptive transformer segmentation framework tailored for multi-institutional Quantitative computed tomography (QCT)<n>Our model is trained and validated on one of the largest hip fracture related research cohorts to date, comprising 1,024 QCT images scans from Tulane University and 384 scans from Rochester, Minnesota for proximal femur segmentation.
  arXiv  Detail & Related papers  (2025-10-30T18:07:56Z)
• A Dual-Task Synergy-Driven Generalization Framework for Pancreatic Cancer Segmentation in CT Scans [10.62594407632477]
  Pancreatic cancer, characterized by its notable prevalence and mortality rates, demands accurate lesion delineation.<n>We propose a generalization framework that synergizes pixel-level classification and regression tasks, to accurately delineate lesions.<n>Our model successfully improves the results of the highly challenging cross-lesion generalized pancreatic cancer segmentation task by 9.51%.
  arXiv  Detail & Related papers  (2025-05-03T00:54:00Z)
• Pre-trained Models Succeed in Medical Imaging with Representation Similarity Degradation [6.545152478351316]
  The study establishes a rigorous problem definition centered on quantifying and analyzing representation similarity trajectories.<n>Our empirical findings reveal the potential existence of high-performance models that preserve both task accuracy and representation similarity to their pre-trained origins.
  arXiv  Detail & Related papers  (2025-03-11T01:37:54Z)
• MIRROR: Multi-Modal Pathological Self-Supervised Representation Learning via Modality Alignment and Retention [57.044719143401664]
  Histopathology and transcriptomics are fundamental modalities in oncology, encapsulating the morphological and molecular aspects of the disease.<n>We present MIRROR, a novel multi-modal representation learning method designed to foster both modality alignment and retention.<n>Extensive evaluations on TCGA cohorts for cancer subtyping and survival analysis highlight MIRROR's superior performance.
  arXiv  Detail & Related papers  (2025-03-01T07:02:30Z)
• Revisiting Adaptive Cellular Recognition Under Domain Shifts: A Contextual Correspondence View [49.03501451546763]
  We identify the importance of implicit correspondences across biological contexts for exploiting domain-invariant pathological composition. We propose self-adaptive dynamic distillation to secure instance-aware trade-offs across different model constituents.
  arXiv  Detail & Related papers  (2024-07-14T04:41:16Z)
• S3M: Scalable Statistical Shape Modeling through Unsupervised Correspondences [91.48841778012782]
  We propose an unsupervised method to simultaneously learn local and global shape structures across population anatomies. Our pipeline significantly improves unsupervised correspondence estimation for SSMs compared to baseline methods. Our method is robust enough to learn from noisy neural network predictions, potentially enabling scaling SSMs to larger patient populations.
  arXiv  Detail & Related papers  (2023-04-15T09:39:52Z)
• Mine yOur owN Anatomy: Revisiting Medical Image Segmentation with Extremely Limited Labels [54.58539616385138]
  We introduce a novel semi-supervised 2D medical image segmentation framework termed Mine yOur owN Anatomy (MONA) First, prior work argues that every pixel equally matters to the model training; we observe empirically that this alone is unlikely to define meaningful anatomical features. Second, we construct a set of objectives that encourage the model to be capable of decomposing medical images into a collection of anatomical features.
  arXiv  Detail & Related papers  (2022-09-27T15:50:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.