Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI

• URL: http://arxiv.org/abs/2606.26894v1
• Date: Thu, 25 Jun 2026 11:26:57 GMT
• Title: Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI
• Abstract summary: We propose a 3D vision transformer that explicitly models both modality-specific representations and cross-modal interactions.<n>We evaluate MICViT on brain age prediction across three heterogeneous datasets.
• Score: 1.0971336662049027
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Brain MRI poses a fundamental challenge for machine learning: models must learn from high-dimensional 3D data spanning multiple co-registered modalities, despite the limited sample sizes typical of neuroimaging studies relative to the diversity in anatomy, pathology, and acquisition conditions. While multimodal imaging provides complementary information critical for clinical interpretation, effectively integrating these signals remains difficult. We propose Multimodal Intra- and Cross-Context Vision Transformer (MICViT), a 3D vision transformer that explicitly models both modality-specific representations and cross-modal interactions across local and global contexts. Concretely, MICViT combines four attention mechanisms: modality-specific local and global attention for intra-modal feature learning, and cross-modal local and global attention to capture interactions between modalities. We evaluate MICViT on brain age prediction across three heterogeneous datasets (UK Biobank, n=41,404; SOOP, n=1,062; Cam-CAN, n=613) using multiple MRI modalities (e.g. T1, FLAIR, DWI, SWI). MICViT consistently outperforms state-of-the-art CNN and transformer baselines in 3D settings. Notably, it benefits more strongly from multimodal inputs, yielding larger performance gains as additional modalities are incorporated. These results demonstrate that explicitly modeling intra- and cross-modal interactions is key to unlocking the full potential of multimodal brain MRI, highlighting a promising direction for representation learning in neuroimaging.

Related papers

• Mr3D-VL: A generalist vision language foundation model for Multiparametric 3D Magnetic Resonance Imaging [3.8658612593287525]
  Mr3D-VL is a visual-language foundation model for multi-parametric 3D MRI.<n>It employs an unsupervised pre-trained shared 3D encoder and 4D rotational positional embedding for dual modality-spatial integration.
  arXiv  Detail & Related papers  (2026-08-13T01:12:34Z)
• Unified Multimodal Model for Brain MRI Imputation and Understanding [19.00815560145768]
  We propose a novel unified multimodal model, UniBrain, for brain magnetic resonance image (MRI) analysis.<n>During training, an interleaved and description-enriched data flow is constructed to train the model in an autoregressive manner.<n>A self-alignment strategy is introduced to leverage dense image embeddings to learn fine-grained anatomical features without requiring detailed image captions.
  arXiv  Detail & Related papers  (2026-06-15T09:51:00Z)
• Modality-Aware and Anatomical Vector-Quantized Autoencoding for Multimodal Brain MRI [15.657081811594418]
  We propose a modality-aware 3D vector-quantized VAE (VQ-VAE) for reconstructing multi-modal brain MRIs.<n>NeuroQuant achieves superior reconstruction fidelity compared to existing VAEs.
  arXiv  Detail & Related papers  (2026-04-06T21:07:34Z)
• Toward a Multi-View Brain Network Foundation Model: Cross-View Consistency Learning Across Arbitrary Atlases [62.33465338932216]
  MV-BrainFM is a multi-view brain network foundation model designed to learn generalizable and scalable representations from brain networks constructed with arbitrary atlases.<n>Experiments on more than 20K subjects from 17 fMRI datasets show that MV-BrainFM consistently outperforms 14 existing brain network foundation models.
  arXiv  Detail & Related papers  (2026-03-20T11:55:00Z)
• Modeling Cross-vision Synergy for Unified Large Vision Model [130.37489011094036]
  PolyV is a unified large vision model that achieves cross-vision synergy at both the architectural and training levels.<n>PolyV consistently outperforms existing models, achieving over 10% average improvement over its backbone.
  arXiv  Detail & Related papers  (2026-03-03T22:44:43Z)
• Towards Generalisable Foundation Models for 3D Brain MRI [5.527537739064968]
  We introduce BrainFound, a self-supervised foundation model for brain MRI built by extending DINO-v2.<n>BrainFound adapts DINO-v2 to model full 3D brain anatomy by incorporating information from sequential MRI slices.<n>It supports both single- and multimodal inputs, enabling a broad range of downstream tasks, including disease detection and image segmentation.
  arXiv  Detail & Related papers  (2025-10-27T15:19:46Z)
• M3Ret: Unleashing Zero-shot Multimodal Medical Image Retrieval via Self-Supervision [24.846428105192405]
  We train M3Ret, a unified visual encoder, without any modality-specific customization.<n>It successfully learns transferable representations using both generative (MAE) and contrastive (SimDINO) self-supervised learning (SSL) paradigms.<n>Our approach sets a new state-of-the-art in zero-shot image-to-image retrieval across all individual modalities, surpassing strong baselines such as DINOv3 and the text-supervised BMC-CLIP.
  arXiv  Detail & Related papers  (2025-09-01T10:59:39Z)
• Mamba Based Feature Extraction And Adaptive Multilevel Feature Fusion For 3D Tumor Segmentation From Multi-modal Medical Image [8.999013226631893]
  Multi-modal 3D medical image segmentation aims to accurately identify tumor regions across different modalities.<n>Traditional convolutional neural network (CNN)-based methods struggle with capturing global features.<n>Transformers-based methods, despite effectively capturing global context, encounter high computational costs in 3D medical image segmentation.
  arXiv  Detail & Related papers  (2025-04-30T03:29:55Z)
• MindFormer: Semantic Alignment of Multi-Subject fMRI for Brain Decoding [50.55024115943266]
  We introduce a novel semantic alignment method of multi-subject fMRI signals using so-called MindFormer. This model is specifically designed to generate fMRI-conditioned feature vectors that can be used for conditioning Stable Diffusion model for fMRI- to-image generation or large language model (LLM) for fMRI-to-text generation. Our experimental results demonstrate that MindFormer generates semantically consistent images and text across different subjects.
  arXiv  Detail & Related papers  (2024-05-28T00:36:25Z)
• SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion Classification Using 3D Multi-Phase Imaging [59.78761085714715]
  This study proposes a novel Siamese Dual-Resolution Transformer (SDR-Former) framework for liver lesion classification. The proposed framework has been validated through comprehensive experiments on two clinical datasets. To support the scientific community, we are releasing our extensive multi-phase MR dataset for liver lesion analysis to the public.
  arXiv  Detail & Related papers  (2024-02-27T06:32:56Z)
• Cross-Modality Deep Feature Learning for Brain Tumor Segmentation [158.8192041981564]
  This paper proposes a novel cross-modality deep feature learning framework to segment brain tumors from the multi-modality MRI data. The core idea is to mine rich patterns across the multi-modality data to make up for the insufficient data scale. Comprehensive experiments are conducted on the BraTS benchmarks, which show that the proposed cross-modality deep feature learning framework can effectively improve the brain tumor segmentation performance.
  arXiv  Detail & Related papers  (2022-01-07T07:46:01Z)
• Hi-Net: Hybrid-fusion Network for Multi-modal MR Image Synthesis [143.55901940771568]
  We propose a novel Hybrid-fusion Network (Hi-Net) for multi-modal MR image synthesis. In our Hi-Net, a modality-specific network is utilized to learn representations for each individual modality. A multi-modal synthesis network is designed to densely combine the latent representation with hierarchical features from each modality.
  arXiv  Detail & Related papers  (2020-02-11T08:26:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.