Partially Supervised Unpaired Multi-Modal Learning for Label-Efficient Medical Image Segmentation

• URL: http://arxiv.org/abs/2503.05190v1
• Date: Fri, 07 Mar 2025 07:22:42 GMT
• Title: Partially Supervised Unpaired Multi-Modal Learning for Label-Efficient Medical Image Segmentation
• Authors: Lei Zhu, Yanyu Xu, Huazhu Fu, Xinxing Xu, Rick Siow Mong Goh, Yong Liu,
• Abstract summary: We term the new learning paradigm as Partially Supervised Unpaired Multi-Modal Learning (PSUMML)<n>We propose a novel Decomposed partial class adaptation with snapshot Ensembled Self-Training (DEST) framework for it.<n>Our framework consists of a compact segmentation network with modality specific normalization layers for learning with partially labeled unpaired multi-modal data.
• Score: 53.723234136550055
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Unpaired Multi-Modal Learning (UMML) which leverages unpaired multi-modal data to boost model performance on each individual modality has attracted a lot of research interests in medical image analysis. However, existing UMML methods require multi-modal datasets to be fully labeled, which incurs tremendous annotation cost. In this paper, we investigate the use of partially labeled data for label-efficient unpaired multi-modal learning, which can reduce the annotation cost by up to one half. We term the new learning paradigm as Partially Supervised Unpaired Multi-Modal Learning (PSUMML) and propose a novel Decomposed partial class adaptation with snapshot Ensembled Self-Training (DEST) framework for it. Specifically, our framework consists of a compact segmentation network with modality specific normalization layers for learning with partially labeled unpaired multi-modal data. The key challenge in PSUMML lies in the complex partial class distribution discrepancy due to partial class annotation, which hinders effective knowledge transfer across modalities. We theoretically analyze this phenomenon with a decomposition theorem and propose a decomposed partial class adaptation technique to precisely align the partially labeled classes across modalities to reduce the distribution discrepancy. We further propose a snapshot ensembled self-training technique to leverage the valuable snapshot models during training to assign pseudo-labels to partially labeled pixels for self-training to boost model performance. We perform extensive experiments under different scenarios of PSUMML for two medical image segmentation tasks, namely cardiac substructure segmentation and abdominal multi-organ segmentation. Our framework outperforms existing methods significantly.

Related papers

• Multimodal Distillation-Driven Ensemble Learning for Long-Tailed Histopathology Whole Slide Images Analysis [16.01677300903562]
  Multiple Instance Learning (MIL) plays a significant role in computational pathology, enabling weakly supervised analysis of Whole Slide Image (WSI) datasets.<n>We propose an ensemble learning method based on MIL, which employs expert decoders with shared aggregators to learn diverse distributions.<n>We introduce a multimodal distillation framework that leverages text encoders pre-trained on pathology-text pairs to distill knowledge.<n>Our method, MDE-MIL, integrates multiple expert branches focusing on specific data distributions to address long-tailed issues.
  arXiv  Detail & Related papers  (2025-03-02T14:31:45Z)
• CLIPErase: Efficient Unlearning of Visual-Textual Associations in CLIP [56.199779065855004]
  We introduce CLIPErase, a novel approach that disentangles and selectively forgets both visual and textual associations. Experiments on the CIFAR-100 and Flickr30K datasets demonstrate that CLIPErase effectively forgets designated associations in zero-shot tasks for multimodal samples.
  arXiv  Detail & Related papers  (2024-10-30T17:51:31Z)
• Versatile Medical Image Segmentation Learned from Multi-Source Datasets via Model Self-Disambiguation [9.068045557591612]
  We propose a cost-effective alternative that harnesses multi-source data with only partial or sparse segmentation labels for training. We devise strategies for model self-disambiguation, prior knowledge incorporation, and imbalance mitigation to tackle challenges associated with inconsistently labeled multi-source data.
  arXiv  Detail & Related papers  (2023-11-17T18:28:32Z)
• Multi-Scale Cross Contrastive Learning for Semi-Supervised Medical Image Segmentation [14.536384387956527]
  We develop a novel Multi-Scale Cross Supervised Contrastive Learning framework to segment structures in medical images. Our approach contrasts multi-scale features based on ground-truth and cross-predicted labels, in order to extract robust feature representations. It outperforms state-of-the-art semi-supervised methods by more than 3.0% in Dice.
  arXiv  Detail & Related papers  (2023-06-25T16:55:32Z)
• Tailored Multi-Organ Segmentation with Model Adaptation and Ensemble [22.82094545786408]
  Multi-organ segmentation is a fundamental task in medical image analysis. Due to expensive labor costs and expertise, the availability of multi-organ annotations is usually limited. We propose a novel dual-stage method that consists of a Model Adaptation stage and a Model Ensemble stage.
  arXiv  Detail & Related papers  (2023-04-14T13:39:39Z)
• MultiTalent: A Multi-Dataset Approach to Medical Image Segmentation [1.146419670457951]
  Current practices limit model training and supervised pre-training to one or a few similar datasets. We propose MultiTalent, a method that leverages multiple CT datasets with diverse and conflicting class definitions.
  arXiv  Detail & Related papers  (2023-03-25T11:37:16Z)
• Toward Unpaired Multi-modal Medical Image Segmentation via Learning Structured Semantic Consistency [24.78258331561847]
  This paper presents a novel scheme to learn the mutual benefits of different modalities to achieve better segmentation results for unpaired medical images. We leverage a carefully designed External Attention Module (EAM) to align semantic class representations and their correlations of different modalities. We have demonstrated the effectiveness of the proposed method on two medical image segmentation scenarios.
  arXiv  Detail & Related papers  (2022-06-21T17:50:29Z)
• Learning from Partially Overlapping Labels: Image Segmentation under Annotation Shift [68.6874404805223]
  We propose several strategies for learning from partially overlapping labels in the context of abdominal organ segmentation. We find that combining a semi-supervised approach with an adaptive cross entropy loss can successfully exploit heterogeneously annotated data.
  arXiv  Detail & Related papers  (2021-07-13T09:22:24Z)
• Few-shot Action Recognition with Prototype-centered Attentive Learning [88.10852114988829]
  Prototype-centered Attentive Learning (PAL) model composed of two novel components. First, a prototype-centered contrastive learning loss is introduced to complement the conventional query-centered learning objective. Second, PAL integrates a attentive hybrid learning mechanism that can minimize the negative impacts of outliers.
  arXiv  Detail & Related papers  (2021-01-20T11:48:12Z)
• Towards Robust Partially Supervised Multi-Structure Medical Image Segmentation on Small-Scale Data [123.03252888189546]
  We propose Vicinal Labels Under Uncertainty (VLUU) to bridge the methodological gaps in partially supervised learning (PSL) under data scarcity. Motivated by multi-task learning and vicinal risk minimization, VLUU transforms the partially supervised problem into a fully supervised problem by generating vicinal labels. Our research suggests a new research direction in label-efficient deep learning with partial supervision.
  arXiv  Detail & Related papers  (2020-11-28T16:31:00Z)
• Unpaired Multi-modal Segmentation via Knowledge Distillation [77.39798870702174]
  We propose a novel learning scheme for unpaired cross-modality image segmentation. In our method, we heavily reuse network parameters, by sharing all convolutional kernels across CT and MRI. We have extensively validated our approach on two multi-class segmentation problems.
  arXiv  Detail & Related papers  (2020-01-06T20:03:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.