Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities

• URL: http://arxiv.org/abs/2604.22177v1
• Date: Fri, 24 Apr 2026 03:02:26 GMT
• Title: Uni-Encoder Meets Multi-Encoders: Representation Before Fusion for Brain Tumor Segmentation with Missing Modalities
• Abstract summary: We propose UniME, a two-stage heterogeneous method for brain tumor segmentation.<n>UniME reconciles the trade-offs among fine-grained structure capture, cross-modal complementarity modeling, and exploitation of available modalities.<n> Experiments on BraTS 2023 and BraTS 2024 show that UniME outperforms previous methods under incomplete multi-modal scenarios.
• Score: 15.253982306329393
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multimodal MRI offers complementary information for brain tumor segmentation, but clinical scans often lack one or more modalities, which degrades segmentation performance. In this paper, we propose UniME (Uni-Encoder Meets Multi-Encoders), a two-stage heterogeneous method for brain tumor segmentation with missing modalities that reconciles the trade-offs among fine-grained structure capture, cross-modal complementarity modeling, and exploitation of available modalities. The idea is to decouple representation learning from segmentation via a two-stage heterogeneous architecture. Stage 1 pretrains a single ViT Uni-Encoder with masked image modeling to establish a unified representation robust to missing modalities. Stage 2 adds modality-specific CNN Multi-Encoders to extract high-resolution, multi-scale, fine-grained features. We fuse these features with the global representation to produce precise segmentations. Experiments on BraTS 2023 and BraTS 2024 show that UniME outperforms previous methods under incomplete multi-modal scenarios. The code is available at https://github.com/Hooorace-S/UniME

Related papers

• Multimodal Visual Surrogate Compression for Alzheimer's Disease Classification [69.87877580725768]
  Multimodal Visual Surrogate Compression (MVSC) learns to compress and adapt large 3D sMRI volumes into compact 2D features.<n>MVSC has two key components: a Volume Context that captures global cross-slice context under textual guidance, and an Adaptive Slice Fusion module that aggregates slice-level information in a text-enhanced, patch-wise manner.
  arXiv  Detail & Related papers  (2026-01-29T13:05:46Z)
• UniMRSeg: Unified Modality-Relax Segmentation via Hierarchical Self-Supervised Compensation [104.59740403500132]
  Multi-modal image segmentation faces real-world deployment challenges from incomplete/corrupted modalities degrading performance.<n>We propose a unified modality-relax segmentation network (UniMRSeg) through hierarchical self-supervised compensation (HSSC)<n>Our approach hierarchically bridges representation gaps between complete and incomplete modalities across input, feature and output levels.
  arXiv  Detail & Related papers  (2025-09-19T17:29:25Z)
• OmniSegmentor: A Flexible Multi-Modal Learning Framework for Semantic Segmentation [74.55725909072903]
  We propose a novel multi-modal learning framework, termed OmniSegmentor.<n>Based on ImageNet, we assemble a large-scale dataset for multi-modal pretraining, called ImageNeXt.<n>We introduce a universal multi-modal pretraining framework that consistently amplifies the model's perceptual capabilities across various scenarios.
  arXiv  Detail & Related papers  (2025-09-18T15:52:44Z)
• XLSTM-HVED: Cross-Modal Brain Tumor Segmentation and MRI Reconstruction Method Using Vision XLSTM and Heteromodal Variational Encoder-Decoder [9.141615533517719]
  We introduce the XLSTM-HVED model, which integrates a heteromodal encoder-decoder framework with the Vision XLSTM module to reconstruct missing MRI modalities.<n>Key innovation of our approach is the Self-Attention Variational (SAVE) module, which improves the integration of modal features.<n>Our experiments using the BraTS 2024 dataset demonstrate that our model significantly outperforms existing advanced methods in handling cases where modalities are missing.
  arXiv  Detail & Related papers  (2024-12-09T09:04:02Z)
• MulModSeg: Enhancing Unpaired Multi-Modal Medical Image Segmentation with Modality-Conditioned Text Embedding and Alternating Training [10.558275557142137]
  We propose a simple Multi-Modal (MulModSeg) strategy to enhance medical image segmentation across multiple modalities. MulModSeg incorporates a modality-conditioned text embedding framework via a frozen text encoder. It consistently outperforms previous methods in segmenting abdominal multiorgan and cardiac substructures for both CT and MR.
  arXiv  Detail & Related papers  (2024-11-23T14:37:01Z)
• Dual-scale Enhanced and Cross-generative Consistency Learning for Semi-supervised Medical Image Segmentation [49.57907601086494]
  Medical image segmentation plays a crucial role in computer-aided diagnosis. We propose a novel Dual-scale Enhanced and Cross-generative consistency learning framework for semi-supervised medical image (DEC-Seg)
  arXiv  Detail & Related papers  (2023-12-26T12:56:31Z)
• M$^{2}$SNet: Multi-scale in Multi-scale Subtraction Network for Medical Image Segmentation [66.89632406480949]
  We propose a general multi-scale in multi-scale subtraction network (M$2$SNet) to finish diverse segmentation from medical image.<n>Our method performs favorably against most state-of-the-art methods under different evaluation metrics on eleven datasets of four different medical image segmentation tasks.
  arXiv  Detail & Related papers  (2023-03-20T06:26:49Z)
• M3AE: Multimodal Representation Learning for Brain Tumor Segmentation with Missing Modalities [29.455215925816187]
  Multimodal magnetic resonance imaging (MRI) provides complementary information for sub-region analysis of brain tumors. It is common to have one or more modalities missing due to image corruption, artifacts, acquisition protocols, allergy to contrast agents, or simply cost. We propose a novel two-stage framework for brain tumor segmentation with missing modalities.
  arXiv  Detail & Related papers  (2023-03-09T14:54:30Z)
• NestedFormer: Nested Modality-Aware Transformer for Brain Tumor Segmentation [29.157465321864265]
  We propose a novel Nested Modality-Aware Transformer (NestedFormer) to explore the intra-modality and inter-modality relationships of multi-modal MRIs for brain tumor segmentation. Built on the transformer-based multi-encoder and single-decoder structure, we perform nested multi-modal fusion for high-level representations of different modalities.
  arXiv  Detail & Related papers  (2022-08-31T14:04:25Z)
• mmFormer: Multimodal Medical Transformer for Incomplete Multimodal Learning of Brain Tumor Segmentation [38.22852533584288]
  We propose a novel Medical Transformer (mmFormer) for incomplete multimodal learning with three main components. The proposed mmFormer outperforms the state-of-the-art methods for incomplete multimodal brain tumor segmentation on almost all subsets of incomplete modalities.
  arXiv  Detail & Related papers  (2022-06-06T08:41:56Z)
• Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation [75.58395328700821]
  We propose a novel model, Multi-modal Gaussian Process Prior Variational Autoencoder (MGP-VAE), to impute one or more missing sub-modalities for a patient scan. MGP-VAE can leverage the Gaussian Process (GP) prior on the Variational Autoencoder (VAE) to utilize the subjects/patients and sub-modalities correlations. We show the applicability of MGP-VAE on brain tumor segmentation where either, two, or three of four sub-modalities may be missing.
  arXiv  Detail & Related papers  (2021-07-07T19:06:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.