Mr3D-VL: A generalist vision language foundation model for Multiparametric 3D Magnetic Resonance Imaging

• URL: http://arxiv.org/abs/2608.12689v1
• Date: Thu, 13 Aug 2026 01:12:34 GMT
• Title: Mr3D-VL: A generalist vision language foundation model for Multiparametric 3D Magnetic Resonance Imaging
• Abstract summary: Mr3D-VL is a visual-language foundation model for multi-parametric 3D MRI.<n>It employs an unsupervised pre-trained shared 3D encoder and 4D rotational positional embedding for dual modality-spatial integration.
• Score: 3.8658612593287525
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-parametric magnetic resonance imaging (mpMRI) is a cornerstone for brain tumor diagnosis and treatment, yet current AI models face critical limitations: their lack of natural language interaction and interpretability impedes spatial information integration and cross-modal reasoning required clinically. Key challenges arise from significant physical meaning differences across modalities, spatial misalignment due to scan intervals, and the need for complex multi-feature interpretation in tasks like glioma grading. While visual-language models (VLMs) show promise in cross-modal understanding, existing methods focus mainly on 2D image modeling, neglecting direct perception of 3D volumetric space. Although 3D VLMs have been proposed for report generation and feature alignment in 3D CT imaging, mpMRI applications demand collaborative inference across multiple imaging modalities-a requirement unmet by current solutions. To address this, we introduce Mr3D-VL, a dedicated visual-language foundation model for multi-parametric 3D MRI. With 4 billion parameters, it employs an unsupervised pre-trained shared 3D encoder and 4D rotational positional embedding for dual modality-spatial integration. Its cross-modal projection layer uses a multi-resolution feature implantation strategy to enhance feature perception across resolutions. Experimental results show significant improvements over existing 4B/7B/30B domain-specific and general-purpose models in text generation tasks, achieving a BERTScore of 0.856 for report generation, with question-answering accuracy at 0.713 and multiple-choice accuracy at 0.912.

Related papers

• Modeling Local, Global, and Cross-Modal Context in Multimodal 3D MRI [1.0971336662049027]
  We propose a 3D vision transformer that explicitly models both modality-specific representations and cross-modal interactions.<n>We evaluate MICViT on brain age prediction across three heterogeneous datasets.
  arXiv  Detail & Related papers  (2026-06-25T11:26:57Z)
• MultiMedVision: Multi-Modal Medical Vision Framework [4.804608924059296]
  We present MultiMedVision, a unified framework for joint 2D/3D representation learning.<n>Our model uses 3D Rotary Positional Embeddings and variable-length sequence packing to process mixed-modality batches.<n>Using a single shared encoder with 5x less data, MultiMedVision achieves competitive performance on both 2D benchmarks.
  arXiv  Detail & Related papers  (2026-05-09T20:29:24Z)
• Multimodal Visual Surrogate Compression for Alzheimer's Disease Classification [69.87877580725768]
  Multimodal Visual Surrogate Compression (MVSC) learns to compress and adapt large 3D sMRI volumes into compact 2D features.<n>MVSC has two key components: a Volume Context that captures global cross-slice context under textual guidance, and an Adaptive Slice Fusion module that aggregates slice-level information in a text-enhanced, patch-wise manner.
  arXiv  Detail & Related papers  (2026-01-29T13:05:46Z)
• MedVL-SAM2: A unified 3D medical vision-language model for multimodal reasoning and prompt-driven segmentation [11.762545584252052]
  We propose a unified 3D medical multimodal model that supports report generation, VQA, and multi-paradigm segmentation.<n>MedVL-SAM2 integrates image-level reasoning and pixel-level perception through a cohesive architecture tailored for 3D medical imaging.<n>Our unified architecture delivers state-of-the-art performance across report generation, VQA, and multiple 3D segmentation tasks.
  arXiv  Detail & Related papers  (2026-01-14T21:21:00Z)
• Towards Generalisable Foundation Models for 3D Brain MRI [5.527537739064968]
  We introduce BrainFound, a self-supervised foundation model for brain MRI built by extending DINO-v2.<n>BrainFound adapts DINO-v2 to model full 3D brain anatomy by incorporating information from sequential MRI slices.<n>It supports both single- and multimodal inputs, enabling a broad range of downstream tasks, including disease detection and image segmentation.
  arXiv  Detail & Related papers  (2025-10-27T15:19:46Z)
• Modality-Aware Feature Matching: A Comprehensive Review of Single- and Cross-Modality Techniques [91.26187560114381]
  Feature matching is a cornerstone task in computer vision, essential for applications such as image retrieval, stereo matching, 3D reconstruction, and SLAM.<n>This survey comprehensively reviews modality-based feature matching, exploring traditional handcrafted methods and contemporary deep learning approaches.
  arXiv  Detail & Related papers  (2025-07-30T15:56:36Z)
• A Unified Model for Compressed Sensing MRI Across Undersampling Patterns [69.19631302047569]
  We propose a unified MRI reconstruction model robust to various measurement undersampling patterns and image resolutions.<n>Our model improves SSIM by 11% and PSNR by 4 dB over a state-of-the-art CNN (End-to-End VarNet) with 600$times$ faster inference than diffusion methods.
  arXiv  Detail & Related papers  (2024-10-05T20:03:57Z)
• Towards Unified 3D Object Detection via Algorithm and Data Unification [70.27631528933482]
  We build the first unified multi-modal 3D object detection benchmark MM- Omni3D and extend the aforementioned monocular detector to its multi-modal version. We name the designed monocular and multi-modal detectors as UniMODE and MM-UniMODE, respectively.
  arXiv  Detail & Related papers  (2024-02-28T18:59:31Z)
• SDR-Former: A Siamese Dual-Resolution Transformer for Liver Lesion Classification Using 3D Multi-Phase Imaging [59.78761085714715]
  This study proposes a novel Siamese Dual-Resolution Transformer (SDR-Former) framework for liver lesion classification. The proposed framework has been validated through comprehensive experiments on two clinical datasets. To support the scientific community, we are releasing our extensive multi-phase MR dataset for liver lesion analysis to the public.
  arXiv  Detail & Related papers  (2024-02-27T06:32:56Z)
• Single-subject Multi-contrast MRI Super-resolution via Implicit Neural Representations [9.683341998041634]
  Implicit Neural Representations (INR) proposed to learn two different contrasts of complementary views in a continuous spatial function. Our model provides realistic super-resolution across different pairs of contrasts in our experiments with three datasets.
  arXiv  Detail & Related papers  (2023-03-27T10:18:42Z)
• Revisiting 3D Context Modeling with Supervised Pre-training for Universal Lesion Detection in CT Slices [48.85784310158493]
  We propose a Modified Pseudo-3D Feature Pyramid Network (MP3D FPN) to efficiently extract 3D context enhanced 2D features for universal lesion detection in CT slices. With the novel pre-training method, the proposed MP3D FPN achieves state-of-the-art detection performance on the DeepLesion dataset. The proposed 3D pre-trained weights can potentially be used to boost the performance of other 3D medical image analysis tasks.
  arXiv  Detail & Related papers  (2020-12-16T07:11:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.