Modality-Agnostic Input Channels Enable Segmentation of Brain lesions in Multimodal MRI with Sequences Unavailable During Training

• URL: http://arxiv.org/abs/2509.09290v1
• Date: Thu, 11 Sep 2025 09:25:30 GMT
• Title: Modality-Agnostic Input Channels Enable Segmentation of Brain lesions in Multimodal MRI with Sequences Unavailable During Training
• Authors: Anthony P. Addison, Felix Wagner, Wentian Xu, Natalie Voets, Konstantinos Kamnitsas,
• Abstract summary: Most segmentation models for multimodal brain MRI are restricted to fixed modalities.<n>Some models generalize to unseen modalities but may lose modality-specific information.<n>This work aims to develop a model that can perform inference on data that contain image modalities unseen during training.
• Score: 4.672822120456554
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Segmentation models are important tools for the detection and analysis of lesions in brain MRI. Depending on the type of brain pathology that is imaged, MRI scanners can acquire multiple, different image modalities (contrasts). Most segmentation models for multimodal brain MRI are restricted to fixed modalities and cannot effectively process new ones at inference. Some models generalize to unseen modalities but may lose discriminative modality-specific information. This work aims to develop a model that can perform inference on data that contain image modalities unseen during training, previously seen modalities, and heterogeneous combinations of both, thus allowing a user to utilize any available imaging modalities. We demonstrate this is possible with a simple, thus practical alteration to the U-net architecture, by integrating a modality-agnostic input channel or pathway, alongside modality-specific input channels. To train this modality-agnostic component, we develop an image augmentation scheme that synthesizes artificial MRI modalities. Augmentations differentially alter the appearance of pathological and healthy brain tissue to create artificial contrasts between them while maintaining realistic anatomical integrity. We evaluate the method using 8 MRI databases that include 5 types of pathologies (stroke, tumours, traumatic brain injury, multiple sclerosis and white matter hyperintensities) and 8 modalities (T1, T1+contrast, T2, PD, SWI, DWI, ADC and FLAIR). The results demonstrate that the approach preserves the ability to effectively process MRI modalities encountered during training, while being able to process new, unseen modalities to improve its segmentation. Project code: https://github.com/Anthony-P-Addison/AGN-MOD-SEG

Related papers

• A Foundation Model for Brain MRI with Dynamic Modality Integration [0.0]
  We present a foundation model for brain MRI that can work with different combinations of imaging sequences.<n>The model uses one encoder with learnable modality embeddings, conditional layer normalization, and a masked autoencoding objective.<n>It is trained on about 60,000 multi-center MRIs using self-supervised reconstruction and modality imputation.
  arXiv  Detail & Related papers  (2025-11-04T21:25:48Z)
• SLaM-DiMM: Shared Latent Modeling for Diffusion Based Missing Modality Synthesis in MRI [0.0]
  Brain MRI scans are often found in four modalities, consisting of T1-weighted with and without contrast enhancement (T1ce and T1w), T2-weighted imaging (T2w) and Flair.<n>We propose SLaM-DiMM, a novel missing modality generation framework that harnesses the power of diffusion models to synthesize any of the four target MRI modalities from other available modalities.
  arXiv  Detail & Related papers  (2025-09-19T14:27:35Z)
• A Modality-agnostic Multi-task Foundation Model for Human Brain Imaging [12.710492824928338]
  We introduce BrainFM, a modality-agnostic, multi-task vision foundation model for human brain imaging.<n>BrainFM is resilient to the appearance of acquired images.<n>It can be directly applied to five fundamental brain imaging tasks.
  arXiv  Detail & Related papers  (2025-08-30T16:15:32Z)
• Multi-modal Vision Pre-training for Medical Image Analysis [11.569448567735435]
  Self-supervised learning has greatly facilitated medical image analysis by suppressing the training data requirement for real-world applications.<n>We conduct a novel multi-modal image pre-training with three proxy tasks to facilitate the learning of cross-modality representations and correlations.<n>Our method is reported in comparison to state-of-the-art pre-training methods, with Dice Score improvement of 0.28%-14.47% across six segmentation benchmarks and a consistent accuracy boost of 0.65%-18.07% in four individual image classification tasks.
  arXiv  Detail & Related papers  (2024-10-14T15:12:16Z)
• Enhancing Incomplete Multi-modal Brain Tumor Segmentation with Intra-modal Asymmetry and Inter-modal Dependency [31.047259264831947]
  A common problem in practice is the unavailability of some modalities due to varying scanning protocols and patient conditions. Previous methods have attempted to address this by fusing accessible multi-modal features, leveraging attention mechanisms, and synthesizing missing modalities. We propose a novel approach that enhances the deep learning-based brain tumor segmentation model from two perspectives.
  arXiv  Detail & Related papers  (2024-06-14T16:54:53Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• NeuroPictor: Refining fMRI-to-Image Reconstruction via Multi-individual Pretraining and Multi-level Modulation [55.51412454263856]
  This paper proposes to directly modulate the generation process of diffusion models using fMRI signals. By training with about 67,000 fMRI-image pairs from various individuals, our model enjoys superior fMRI-to-image decoding capacity.
  arXiv  Detail & Related papers  (2024-03-27T02:42:52Z)
• Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Grading [47.50733518140625]
  Brain tumor represents one of the most fatal cancers around the world, and is very common in children and the elderly. We propose a novel cross-modality guidance-aided multi-modal learning with dual attention for addressing the task of MRI brain tumor grading.
  arXiv  Detail & Related papers  (2024-01-17T07:54:49Z)
• fMRI-PTE: A Large-scale fMRI Pretrained Transformer Encoder for Multi-Subject Brain Activity Decoding [54.17776744076334]
  We propose fMRI-PTE, an innovative auto-encoder approach for fMRI pre-training. Our approach involves transforming fMRI signals into unified 2D representations, ensuring consistency in dimensions and preserving brain activity patterns. Our contributions encompass introducing fMRI-PTE, innovative data transformation, efficient training, a novel learning strategy, and the universal applicability of our approach.
  arXiv  Detail & Related papers  (2023-11-01T07:24:22Z)
• DIGEST: Deeply supervIsed knowledGE tranSfer neTwork learning for brain tumor segmentation with incomplete multi-modal MRI scans [16.93394669748461]
  Brain tumor segmentation based on multi-modal magnetic resonance imaging (MRI) plays a pivotal role in assisting brain cancer diagnosis, treatment, and postoperative evaluations. Despite the achieved inspiring performance by existing automatic segmentation methods, multi-modal MRI data are still unavailable in real-world clinical applications. We propose a Deeply supervIsed knowledGE tranSfer neTwork (DIGEST), which achieves accurate brain tumor segmentation under different modality-missing scenarios.
  arXiv  Detail & Related papers  (2022-11-15T09:01:14Z)
• SMU-Net: Style matching U-Net for brain tumor segmentation with missing modalities [4.855689194518905]
  We propose a style matching U-Net (SMU-Net) for brain tumour segmentation on MRI images. Our co-training approach utilizes a content and style-matching mechanism to distill the informative features from the full-modality network into a missing modality network. Our style matching module adaptively recalibrates the representation space by learning a matching function to transfer the informative and textural features from a full-modality path into a missing-modality path.
  arXiv  Detail & Related papers  (2022-04-06T17:55:19Z)
• Cross-Modality Deep Feature Learning for Brain Tumor Segmentation [158.8192041981564]
  This paper proposes a novel cross-modality deep feature learning framework to segment brain tumors from the multi-modality MRI data. The core idea is to mine rich patterns across the multi-modality data to make up for the insufficient data scale. Comprehensive experiments are conducted on the BraTS benchmarks, which show that the proposed cross-modality deep feature learning framework can effectively improve the brain tumor segmentation performance.
  arXiv  Detail & Related papers  (2022-01-07T07:46:01Z)
• Modality Completion via Gaussian Process Prior Variational Autoencoders for Multi-Modal Glioma Segmentation [75.58395328700821]
  We propose a novel model, Multi-modal Gaussian Process Prior Variational Autoencoder (MGP-VAE), to impute one or more missing sub-modalities for a patient scan. MGP-VAE can leverage the Gaussian Process (GP) prior on the Variational Autoencoder (VAE) to utilize the subjects/patients and sub-modalities correlations. We show the applicability of MGP-VAE on brain tumor segmentation where either, two, or three of four sub-modalities may be missing.
  arXiv  Detail & Related papers  (2021-07-07T19:06:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.