A Vision-Language Foundation Model for Precise and Comprehensive Brain Tumor Diagnosis from Preoperative Multimodal Data

• URL: http://arxiv.org/abs/2609.16597v1
• Date: Tue, 15 Sep 2026 03:46:03 GMT
• Title: A Vision-Language Foundation Model for Precise and Comprehensive Brain Tumor Diagnosis from Preoperative Multimodal Data
• Abstract summary: We developed BrainVLM to classify all 12 World Health Organization (WHO) 2021 brain tumor types.<n>BrainVLM was trained on multi-modal data (MRI scans, demographics, and radiology reports) from 40,043 individuals.<n>It was validated on 5,211 patients with pathologically confirmed brain tumors.
• Score: 50.52606847963128
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Background Non-invasive presurgical diagnosis of brain tumor types from Magnetic Resonance Imaging (MRI) is essential but challenging due to overlapping imaging features across tumor types, inter-observer variability, and the extensive training required for expertise. We aimed to develop an MRI-based Artificial Intelligence (AI) model for automatic and reliable brain tumor classification with diagnostic uncertainty quantification and radiology reports generation. Methods We developed BrainVLM to classify all 12 World Health Organization (WHO) 2021 brain tumor types. BrainVLM integrates an uncertainty quantification strategy to indicate prediction reliability and a module for generating radiology reports to elucidate the clinical rationale. BrainVLM was trained on multi-modal data (MRI scans, demographics, and radiology reports) from 40,043 individuals. It was validated on 5,211 patients with pathologically confirmed brain tumors, including 3,877 held-out patients from the primary hospital and 1,334 patients from 11 independent hospitals. We further conducted two proof-of-concept studies to validate its clinical utility in AI-clinician workflows: 1) a blinded multi-reader study where 12 neuroradiologists across varying experience levels interpreted 248 retrospective cases with or without AI assistance, and 2) a real-world prospective study in which 1,009 patients were independently and blindly assessed by BrainVLM and radiologists before surgery. Additionally, we demonstrated BrainVLM's utility in preoperative molecular subgroup prediction for adult-type diffuse gliomas, using a multi-center cohort of 632 patients.

Related papers

• Towards Label-Free Brain Tumor Segmentation: Unsupervised Learning with Multimodal MRI [7.144319861722029]
  Unsupervised anomaly detection (UAD) presents a complementary alternative to supervised learning for brain tumor segmentation in MRI.<n>We propose a novel Multimodal Vision Transformer Autoencoder (MViT-AE) trained exclusively on healthy brain MRIs to detect and localize tumors.<n>Our method achieves clinically meaningful tumor localization, with lesion-wise Dice Similarity Coefficient of 0.437 (Whole Tumor), 0.316 (Tumor Core), and 0.350 (Enhancing Tumor) on the test set, and an anomaly Detection Rate of 89.4% on the validation set.
  arXiv  Detail & Related papers  (2025-10-17T14:26:30Z)
• Towards a general-purpose foundation model for fMRI analysis [58.06455456423138]
  We introduce NeuroSTORM, a framework that learns from 4D fMRI volumes and enables efficient knowledge transfer across diverse applications.<n>NeuroSTORM is pre-trained on 28.65 million fMRI frames (>9,000 hours) from over 50,000 subjects across multiple centers and ages 5 to 100.<n>It outperforms existing methods across five tasks: age/gender prediction, phenotype prediction, disease diagnosis, fMRI-to-image retrieval, and task-based fMRI.
  arXiv  Detail & Related papers  (2025-06-11T23:51:01Z)
• Real-Time Brain Tumor Detection in Intraoperative Ultrasound Using YOLO11: From Model Training to Deployment in the Operating Room [13.503074690080453]
  Intraoperative ultrasound (ioUS) is a valuable tool in brain tumor surgery due to its versatility, affordability, and seamless integration into the surgical workflow.<n>This study aimed to enhance the interpretability of ioUS images by developing a real-time brain tumor detection system deployable in the operating room.<n>We collected 2D ioUS images from the Brain Tumor Intraoperative Database (BraTioUS) and the public ReMIND dataset, annotated with expert-refined tumor labels.<n>Using the YOLO11 architecture and its variants, we trained object detection models to identify brain tumors.
  arXiv  Detail & Related papers  (2025-01-27T12:29:19Z)
• SKIPNet: Spatial Attention Skip Connections for Enhanced Brain Tumor Classification [3.8233569758620063]
  Early detection of brain tumors is essential for timely treatment, yet access to diagnostic facilities remains limited in remote areas.<n>This study proposes an automated deep learning model for brain tumor detection and classification using MRI data.<n>The model, incorporating spatial attention, achieved 96.90% accuracy, enhancing the aggregation of contextual information for better pattern recognition.
  arXiv  Detail & Related papers  (2024-12-10T18:32:42Z)
• Magnetic Resonance Imaging Feature-Based Subtyping and Model Ensemble for Enhanced Brain Tumor Segmentation [6.14919256198409]
  We propose a deep learning-based ensemble approach that integrates state-of-the-art segmentation models.<n>Given the heterogeneous nature of the tumors present in the BraTS datasets, this approach enhances the precision and generalizability of segmentation models.
  arXiv  Detail & Related papers  (2024-12-05T12:00:00Z)
• MGH Radiology Llama: A Llama 3 70B Model for Radiology [50.42811030970618]
  This paper presents an advanced radiology-focused large language model: MGH Radiology Llama.<n>It is developed using the Llama 3 70B model, building upon previous domain-specific models like Radiology-GPT and Radiology-Llama2.<n>Our evaluation, incorporating both traditional metrics and a GPT-4-based assessment, highlights the enhanced performance of this work over general-purpose LLMs.
  arXiv  Detail & Related papers  (2024-08-13T01:30:03Z)
• Analysis of the 2024 BraTS Meningioma Radiotherapy Planning Automated Segmentation Challenge [45.3253187215396]
  The 2024 Brain Tumor Meningioma Radiotherapy (BraTS-MEN-RT) challenge aimed to advance automated segmentation algorithms.<n>We describe the design and results from the BraTS-MEN-RT challenge.
  arXiv  Detail & Related papers  (2024-05-28T17:25:43Z)
• Cross-modality Guidance-aided Multi-modal Learning with Dual Attention for MRI Brain Tumor Grading [47.50733518140625]
  Brain tumor represents one of the most fatal cancers around the world, and is very common in children and the elderly. We propose a novel cross-modality guidance-aided multi-modal learning with dual attention for addressing the task of MRI brain tumor grading.
  arXiv  Detail & Related papers  (2024-01-17T07:54:49Z)
• UniBrain: Universal Brain MRI Diagnosis with Hierarchical Knowledge-enhanced Pre-training [66.16134293168535]
  We propose a hierarchical knowledge-enhanced pre-training framework for the universal brain MRI diagnosis, termed as UniBrain. Specifically, UniBrain leverages a large-scale dataset of 24,770 imaging-report pairs from routine diagnostics.
  arXiv  Detail & Related papers  (2023-09-13T09:22:49Z)
• Patched Diffusion Models for Unsupervised Anomaly Detection in Brain MRI [55.78588835407174]
  We propose a method that reformulates the generation task of diffusion models as a patch-based estimation of healthy brain anatomy. We evaluate our approach on data of tumors and multiple sclerosis lesions and demonstrate a relative improvement of 25.1% compared to existing baselines.
  arXiv  Detail & Related papers  (2023-03-07T09:40:22Z)
• Deep Convolutional Neural Networks Model-based Brain Tumor Detection in Brain MRI Images [0.0]
  Our work involves implementing a deep convolutional neural network (DCNN) for diagnosing brain tumors from MR images. Our model can single out the MR images with tumors with an overall accuracy of 96%.
  arXiv  Detail & Related papers  (2020-10-03T07:42:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.