MCSeg: Pre-training and Fine-tuning Volumetric Pyramid Transformer for Multi-modal Cardiac Image Segmentation

• URL: http://arxiv.org/abs/2608.30371v1
• Date: Mon, 31 Aug 2026 07:27:10 GMT
• Title: MCSeg: Pre-training and Fine-tuning Volumetric Pyramid Transformer for Multi-modal Cardiac Image Segmentation
• Abstract summary: We introduce MCSeg, a volumetric transformer-based network tailored for multi-modal cardiac segmentation.<n>Unlike conventional skip connections, the SFP effectively bridges the single-scale 3D Vision Transformer (ViT) encoder and the multi-scale CNN decoder.<n>In experiments, MCSeg consistently outperforms eleven SOTA methods on CT dataset ImageCHD, multi-modal dataset MM-WHS, MRI dataset HVSMR-2.0 and MSD Heart.
• Score: 15.017233517707025
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic cardiac image segmentation is pivotal for diagnosing and treating cardiac diseases. In this work, we introduce MCSeg, a volumetric transformer-based network tailored for multi-modal cardiac segmentation. To overcome the architectural mismatch inherent in existing hybrid networks, we propose a novel Scaling Feature Pyramid (SFP). Unlike conventional skip connections, the SFP effectively bridges the single-scale 3D Vision Transformer (ViT) encoder and the multi-scale CNN decoder by transforming the ViT's output into a hierarchical feature pyramid, ensuring that global contextual information is effectively leveraged. For the training paradigm, the ViT encoder first undergoes self-supervised pre-training via masked image modeling. Subsequently, the network is fine-tuned on downstream tasks, during which a regional mutual information (RMI) loss is integrated to improve boundary segmentation accuracy. In experiments, MCSeg consistently outperforms eleven SOTA methods on CT dataset ImageCHD, multi-modal dataset MM-WHS, MRI dataset HVSMR-2.0 and MSD Heart, highlighting the effectiveness of our MCSeg for multi-modal cardiac segmentation tasks. Furthermore, MCSeg's superior performance in few-shot experiment showcases its significant potential in adapting to limited data scenarios. Codes and pre-trained ViT-B weights are open-sourced at https://openi.pcl.ac.cn/OpenMedIA/MCSeg

Related papers

• MultiMAE for Brain MRIs: Robustness to Missing Inputs Using Multi-Modal Masked Autoencoder [18.774351784192266]
  Missing input sequences are common in medical imaging data, posing a challenge for deep learning models reliant on complete input data.<n>We develop a masked autoencoder (MAE) paradigm for multi-modal, multi-task learning in 3D medical imaging with brain MRIs.
  arXiv  Detail & Related papers  (2025-09-14T21:33:59Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• Hi-End-MAE: Hierarchical encoder-driven masked autoencoders are stronger vision learners for medical image segmentation [21.183229457060634]
  We pre-train Hi-End-MAE on a large-scale dataset of 10K CT scans and evaluate its performance across seven public medical image segmentation benchmarks.<n>Hi-End-MAE achieves superior transfer learning capabilities across various downstream tasks, revealing the potential of ViT in medical imaging applications.
  arXiv  Detail & Related papers  (2025-02-12T12:14:02Z)
• MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training [10.558275557142137]
  We propose a simple Multi-Modal (MulModSeg) strategy to enhance medical image segmentation across multiple modalities. MulModSeg incorporates a modality-conditioned text embedding framework via a frozen text encoder. It consistently outperforms previous methods in segmenting abdominal multiorgan and cardiac substructures for both CT and MR.
  arXiv  Detail & Related papers  (2024-11-23T14:37:01Z)
• Enhanced Masked Image Modeling to Avoid Model Collapse on Multi-modal MRI Datasets [6.3467517115551875]
  Masked image modeling (MIM) has shown promise in utilizing unlabeled data.<n>We analyze and address model collapse in two types: complete collapse and dimensional collapse.<n>We construct the enhanced MIM (E-MIM) with HMP and PBT module to avoid model collapse multi-modal MRI.
  arXiv  Detail & Related papers  (2024-07-15T01:11:30Z)
• SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion Classification Using 3D Multi-Phase Imaging [59.78761085714715]
  This study proposes a novel Siamese Dual-Resolution Transformer (SDR-Former) framework for liver lesion classification. The proposed framework has been validated through comprehensive experiments on two clinical datasets. To support the scientific community, we are releasing our extensive multi-phase MR dataset for liver lesion analysis to the public.
  arXiv  Detail & Related papers  (2024-02-27T06:32:56Z)
• Disruptive Autoencoders: Leveraging Low-level features for 3D Medical Image Pre-training [51.16994853817024]
  This work focuses on designing an effective pre-training framework for 3D radiology images. We introduce Disruptive Autoencoders, a pre-training framework that attempts to reconstruct the original image from disruptions created by a combination of local masking and low-level perturbations. The proposed pre-training framework is tested across multiple downstream tasks and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-07-31T17:59:42Z)
• Multi-scale Hierarchical Vision Transformer with Cascaded Attention Decoding for Medical Image Segmentation [8.530680502975095]
  We introduce a Multi-scale hiERarchical vIsion Transformer (MERIT) backbone network, which improves the generalizability of the model by computing SA at multiple scales. We also incorporate an attention-based decoder, namely Cascaded Attention Decoding (CASCADE), for further refinement of multi-stage features generated by MERIT.
  arXiv  Detail & Related papers  (2023-03-29T17:58:40Z)
• M$^{2}$SNet: Multi-scale in Multi-scale Subtraction Network for Medical Image Segmentation [66.89632406480949]
  We propose a general multi-scale in multi-scale subtraction network (M$2$SNet) to finish diverse segmentation from medical image.<n>Our method performs favorably against most state-of-the-art methods under different evaluation metrics on eleven datasets of four different medical image segmentation tasks.
  arXiv  Detail & Related papers  (2023-03-20T06:26:49Z)
• Multi-scale Transformer Network with Edge-aware Pre-training for Cross-Modality MR Image Synthesis [52.41439725865149]
  Cross-modality magnetic resonance (MR) image synthesis can be used to generate missing modalities from given ones. Existing (supervised learning) methods often require a large number of paired multi-modal data to train an effective synthesis model. We propose a Multi-scale Transformer Network (MT-Net) with edge-aware pre-training for cross-modality MR image synthesis.
  arXiv  Detail & Related papers  (2022-12-02T11:40:40Z)
• Dual Multi-scale Mean Teacher Network for Semi-supervised Infection Segmentation in Chest CT Volume for COVID-19 [76.51091445670596]
  Automated detecting lung infections from computed tomography (CT) data plays an important role for combating COVID-19. Most current COVID-19 infection segmentation methods mainly relied on 2D CT images, which lack 3D sequential constraint. Existing 3D CT segmentation methods focus on single-scale representations, which do not achieve the multiple level receptive field sizes on 3D volume.
  arXiv  Detail & Related papers  (2022-11-10T13:11:21Z)
• ECG Heartbeat Classification Using Multimodal Fusion [13.524306011331303]
  We propose two computationally efficient multimodal fusion frameworks for ECG heart beat classification. In MFF, we extracted features from penultimate layer of CNNs and fused them to get unique and interdependent information. We achieved classification accuracy of 99.7% and 99.2% on arrhythmia and MI classification, respectively.
  arXiv  Detail & Related papers  (2021-07-21T03:48:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.