Breast Cancer VLMs: Clinically Practical Vision-Language Train-Inference Models

• URL: http://arxiv.org/abs/2510.25051v1
• Date: Wed, 29 Oct 2025 00:37:18 GMT
• Title: Breast Cancer VLMs: Clinically Practical Vision-Language Train-Inference Models
• Authors: Shunjie-Fabian Zheng, Hyeonjun Lee, Thijs Kooi, Ali Diba,
• Abstract summary: This study introduces a novel framework that combines visual features from 2D mammograms with structured textual descriptors derived from easily accessible clinical metadata.<n>Our proposed methods in this study demonstrate that strategic integration of convolutional neural networks (ConvNets) with language representations achieves superior performance to vision transformer-based models.
• Score: 2.7165660672916787
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Breast cancer remains the most commonly diagnosed malignancy among women in the developed world. Early detection through mammography screening plays a pivotal role in reducing mortality rates. While computer-aided diagnosis (CAD) systems have shown promise in assisting radiologists, existing approaches face critical limitations in clinical deployment - particularly in handling the nuanced interpretation of multi-modal data and feasibility due to the requirement of prior clinical history. This study introduces a novel framework that synergistically combines visual features from 2D mammograms with structured textual descriptors derived from easily accessible clinical metadata and synthesized radiological reports through innovative tokenization modules. Our proposed methods in this study demonstrate that strategic integration of convolutional neural networks (ConvNets) with language representations achieves superior performance to vision transformer-based models while handling high-resolution images and enabling practical deployment across diverse populations. By evaluating it on multi-national cohort screening mammograms, our multi-modal approach achieves superior performance in cancer detection and calcification identification compared to unimodal baselines, with particular improvements. The proposed method establishes a new paradigm for developing clinically viable VLM-based CAD systems that effectively leverage imaging data and contextual patient information through effective fusion mechanisms.

Related papers

• A Semantically Enhanced Generative Foundation Model Improves Pathological Image Synthesis [82.01597026329158]
  We introduce a Correlation-Regulated Alignment Framework for Tissue Synthesis (CRAFTS) for pathology-specific text-to-image synthesis.<n>CRAFTS incorporates a novel alignment mechanism that suppresses semantic drift to ensure biological accuracy.<n>This model generates diverse pathological images spanning 30 cancer types, with quality rigorously validated by objective metrics and pathologist evaluations.
  arXiv  Detail & Related papers  (2025-12-15T10:22:43Z)
• OncoVision: Integrating Mammography and Clinical Data through Attention-Driven Multimodal AI for Enhanced Breast Cancer Diagnosis [0.7998211927101394]
  OncoVision is a multimodal AI pipeline that combines mammography images and clinical data for better breast cancer diagnosis.<n>It jointly segments four ROIs - masses, calcifications, axillary findings, and breast tissues.<n>It robustly predicts ten structured clinical features: mass morphology, calcification type, ACR breast density, and BI-RADS categories.
  arXiv  Detail & Related papers  (2025-11-24T20:04:26Z)
• Integrating AI for Human-Centric Breast Cancer Diagnostics: A Multi-Scale and Multi-View Swin Transformer Framework [5.211860566766601]
  The paper focuses on the integration of AI within a Human-Centric workflow to enhance breast cancer diagnostics.<n>We propose a hybrid, multi-scale and multi-view Swin Transformer-based framework (MSMV-Swin) that enhances diagnostic robustness and accuracy.
  arXiv  Detail & Related papers  (2025-03-17T15:48:56Z)
• A Generative Framework for Bidirectional Image-Report Understanding in Chest Radiography [1.2289361708127877]
  Multi-Stage Adaptive Vision-Language Tuning (MAViLT) is a novel framework designed to enhance multimodal reasoning and generation for vision-based understanding.<n>MAViLT incorporates a clinical gradient-weighted tokenization process and a hierarchical fine-tuning strategy, enabling it to generate accurate radiology reports, synthesize realistic CXRs from text, and answer vision-based clinical questions.<n>We evaluate MAViLT on two benchmark datasets, MIMIC-CXR and Indiana University CXR, achieving state-of-the-art results across all tasks.
  arXiv  Detail & Related papers  (2025-02-09T15:02:57Z)
• Clinical Evaluation of Medical Image Synthesis: A Case Study in Wireless Capsule Endoscopy [63.39037092484374]
  Synthetic Data Generation based on Artificial Intelligence (AI) can transform the way clinical medicine is delivered.<n>This study focuses on the clinical evaluation of medical SDG, with a proof-of-concept investigation on diagnosing Inflammatory Bowel Disease (IBD) using Wireless Capsule Endoscopy (WCE) images.<n>The results show that TIDE-II generates clinically plausible, very realistic WCE images, of improved quality compared to relevant state-of-the-art generative models.
  arXiv  Detail & Related papers  (2024-10-31T19:48:50Z)
• MoRE: Multi-Modal Contrastive Pre-training with Transformers on X-Rays, ECGs, and Diagnostic Report [4.340464264725625]
  We introduce a novel Multi-Modal Contrastive Pre-training Framework that synergistically combines X-rays, electrocardiograms (ECGs) and radiology/cardiology reports. We utilize LoRA-Peft to significantly reduce trainable parameters in the LLM and incorporate recent linear attention dropping strategy in the Vision Transformer(ViT) for smoother attention. To the best of our knowledge, we are the first to propose an integrated model that combines X-ray, ECG, and Radiology/Cardiology Report with this approach.
  arXiv  Detail & Related papers  (2024-10-21T17:42:41Z)
• Advancing Histopathology-Based Breast Cancer Diagnosis: Insights into Multi-Modality and Explainability [2.8145472964232137]
  Using multi-modal techniques, integrating both image and non-image data, marks a transformative advancement in breast cancer diagnosis. This review utilizes multi-modal data and emphasizes explainability to enhance diagnostic accuracy, clinician confidence, and patient engagement.
  arXiv  Detail & Related papers  (2024-06-07T19:23:22Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• Polar-Net: A Clinical-Friendly Model for Alzheimer's Disease Detection in OCTA Images [53.235117594102675]
  Optical Coherence Tomography Angiography is a promising tool for detecting Alzheimer's disease (AD) by imaging the retinal microvasculature. We propose a novel deep-learning framework called Polar-Net to provide interpretable results and leverage clinical prior knowledge. We show that Polar-Net outperforms existing state-of-the-art methods and provides more valuable pathological evidence for the association between retinal vascular changes and AD.
  arXiv  Detail & Related papers  (2023-11-10T11:49:49Z)
• LLM-driven Multimodal Target Volume Contouring in Radiation Oncology [46.23891509553877]
  Large language models (LLMs) can facilitate the integration of the textural information and images. We present a novel LLM-driven multimodal AI, namely LLMSeg, that is applicable to the challenging task of target volume contouring for radiation therapy. We demonstrate that the proposed model exhibits markedly improved performance compared to conventional unimodal AI models.
  arXiv  Detail & Related papers  (2023-11-03T13:38:42Z)
• BI-RADS-Net: An Explainable Multitask Learning Approach for Cancer Diagnosis in Breast Ultrasound Images [69.41441138140895]
  This paper introduces BI-RADS-Net, a novel explainable deep learning approach for cancer detection in breast ultrasound images. The proposed approach incorporates tasks for explaining and classifying breast tumors, by learning feature representations relevant to clinical diagnosis. Explanations of the predictions (benign or malignant) are provided in terms of morphological features that are used by clinicians for diagnosis and reporting in medical practice.
  arXiv  Detail & Related papers  (2021-10-05T19:14:46Z)
• Explaining Clinical Decision Support Systems in Medical Imaging using Cycle-Consistent Activation Maximization [112.2628296775395]
  Clinical decision support using deep neural networks has become a topic of steadily growing interest. clinicians are often hesitant to adopt the technology because its underlying decision-making process is considered to be intransparent and difficult to comprehend. We propose a novel decision explanation scheme based on CycleGAN activation which generates high-quality visualizations of classifier decisions even in smaller data sets.
  arXiv  Detail & Related papers  (2020-10-09T14:39:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.