BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine

• URL: http://arxiv.org/abs/2606.04911v1
• Date: Wed, 03 Jun 2026 14:06:52 GMT
• Title: BreastGPT: A Multimodal Large Language Model for the Full Spectrum of Breast Cancer Clinical Routine
• Abstract summary: We introduce textbfBreastStage, a workflow-aligned breast imaging instruction corpus.<n>We then build textbfBreastGPT, a unified MLLM equipped with a dual-branch visual encoder and concept-preserving token compression.<n>On BreastStage-Bench, BreastGPT achieves 75.66% closed-ended accuracy and 89.92% open-ended score, outperforming both general-purpose and medical-specific MLLMs.
• Score: 18.209554692764733
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Breast cancer remains a leading cause of cancer-related mortality among women. Its clinical management requires multimodal reasoning across a clinical workflow that spans \textit{screening}, \textit{diagnosis} and \textit{treatment planning}, where each stage involves distinct imaging modalities, task objectives, and reasoning patterns. However, constrained by data scarcity and model versatility, existing medical MLLMs are typically evaluated on isolated modalities or narrow task families, limiting their ability to support workflow-level clinical reasoning. In this work, we first introduce \textbf{BreastStage}, a workflow-aligned breast imaging instruction corpus comprising 1.86M instruction-following pairs curated from 17 sub-datasets across 5 imaging modalities and 136 task templates. Its held-out split, \textbf{BreastStage-Bench}, provides a comprehensive benchmark for evaluating multimodal reasoning across the breast cancer care continuum. Building on this corpus, we propose \textbf{BreastGPT}, a unified MLLM equipped with a dual-branch visual encoder and concept-preserving token compression to bridge the scale gap between standard radiology and gigapixel pathology. On BreastStage-Bench, BreastGPT achieves 75.66\% closed-ended accuracy and 89.92\% open-ended score, outperforming both general-purpose and medical-specific MLLMs across clinical stages and task formats. These results suggest that workflow-aligned data and cross-scale visual modeling are critical for clinically grounded medical MLLMs. All data, code, and model checkpoints are released at https://yangyy-liu.github.io/BreastGPT.io.

Related papers

• MeisenMeister: A Simple Two Stage Pipeline for Breast Cancer Classification on MRI [3.2340954609207646]
  The ODELIA Breast MRI Challenge 2025 addresses a critical issue in breast cancer screening.<n>This write-up provides a comprehensive overview of our approach to the challenge.
  arXiv  Detail & Related papers  (2025-10-31T09:53:59Z)
• Epistemic-aware Vision-Language Foundation Model for Fetal Ultrasound Interpretation [83.02147613524032]
  We introduce FetalMind, a medical AI system tailored to fetal ultrasound for both report generation and diagnosis.<n>We propose Salient Epistemic Disentanglement (SED), which injects an expert-curated bipartite graph into the model to decouple view-disease associations.<n>FetalMind outperforms open- and closed-source baselines across all gestational stages, achieving +14% average gains and +61.2% higher accuracy on critical conditions.
  arXiv  Detail & Related papers  (2025-10-14T19:57:03Z)
• A Versatile Foundation Model for AI-enabled Mammogram Interpretation [36.48796962019666]
  VersaMammo is a versatile foundation model for mammograms designed to overcome limitations.<n>First, a teacher model is trained via self-supervised learning to extract transferable features from unlabeled mammograms.<n>Then, supervised learning combined with knowledge distillation transfers both features and clinical knowledge into VersaMammo.
  arXiv  Detail & Related papers  (2025-09-24T16:04:39Z)
• Cross-Attention Multimodal Fusion for Breast Cancer Diagnosis: Integrating Mammography and Clinical Data with Explainability [0.0]
  Current computer-aided systems only use characteristics from mammograms.<n>Will clinical features greatly enhance the categorisation of breast lesions?<n>In what ways may explainable AI approaches improve the interpretability and reliability of models used to diagnose breast cancer?
  arXiv  Detail & Related papers  (2025-08-21T23:23:06Z)
• Mammo-SAE: Interpreting Breast Cancer Concept Learning with Sparse Autoencoders [7.23389716633927]
  Interpretability is critical in high-stakes domains such as medical imaging.<n>We introduce Sparse Autoencoder (SAE)-based interpretability to breast imaging.
  arXiv  Detail & Related papers  (2025-07-21T03:59:21Z)
• ChestX-Reasoner: Advancing Radiology Foundation Models with Reasoning through Step-by-Step Verification [57.22053411719822]
  ChestX-Reasoner is a radiology diagnosis MLLM designed to leverage process supervision mined directly from clinical reports.<n>Our two-stage training framework combines supervised fine-tuning and reinforcement learning guided by process rewards to better align model reasoning with clinical standards.
  arXiv  Detail & Related papers  (2025-04-29T16:48:23Z)
• Pathological Prior-Guided Multiple Instance Learning For Mitigating Catastrophic Forgetting in Breast Cancer Whole Slide Image Classification [50.899861205016265]
  We propose a new framework PaGMIL to mitigate catastrophic forgetting in breast cancer WSI classification.<n>Our framework introduces two key components into the common MIL model architecture.<n>We evaluate the continual learning performance of PaGMIL across several public breast cancer datasets.
  arXiv  Detail & Related papers  (2025-03-08T04:51:58Z)
• End-to-End Breast Cancer Radiotherapy Planning via LMMs with Consistency Embedding [47.360760580820966]
  We present RO-LMM, a comprehensive large multimodal model (LMM) tailored for the field of radiation oncology.<n>This model effectively manages a series of tasks within the clinical workflow, including clinical context summarization, radiation treatment plan suggestion, and plan-guided target volume segmentation.<n>We present a novel Consistency Embedding Fine-Tuning (CEFTune) technique, which boosts LMM's robustness to noisy inputs while preserving the consistency of handling clean inputs.
  arXiv  Detail & Related papers  (2023-11-27T14:49:06Z)
• Advancing Radiograph Representation Learning with Masked Record Modeling [52.04899592688968]
  We formulate the self- and report-completion as two complementary objectives and present a unified framework based on masked record modeling (MRM) MRM reconstructs masked image patches and masked report tokens following a multi-task scheme to learn knowledge-enhanced semantic representations. Specifically, we find that MRM offers superior performance in label-efficient fine-tuning.
  arXiv  Detail & Related papers  (2023-01-30T18:33:32Z)
• Multimodal Breast Lesion Classification Using Cross-Attention Deep Networks [0.08635315042809139]
  Most computer-aided systems rely solely on mammogram features to classify breast lesions. This paper investigates several multimodal deep networks based on feature concatenation, cross-attention, and co-attention to combine mammograms and categorical clinical variables.
  arXiv  Detail & Related papers  (2021-08-21T23:01:31Z)
• Two-stage multi-scale breast mass segmentation for full mammogram analysis without user intervention [2.7490008316742096]
  Mammography is the primary imaging modality used for early detection and diagnosis of breast cancer. Among diverse types of breast abnormalities, masses are the most important clinical findings of breast carcinomas. We present a two-stage multi-scale pipeline that provides accurate mass contours from high-resolution full mammograms.
  arXiv  Detail & Related papers  (2020-02-27T13:16:22Z)
• Stan: Small tumor-aware network for breast ultrasound image segmentation [68.8204255655161]
  We propose a novel deep learning architecture called Small Tumor-Aware Network (STAN) to improve the performance of segmenting tumors with different size. The proposed approach outperformed the state-of-the-art approaches in segmenting small breast tumors.
  arXiv  Detail & Related papers  (2020-02-03T22:25:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.