A Specialized Large Multimodal Model for Interpreting PET/CT in Head and Neck Cancer

• URL: http://arxiv.org/abs/2609.05532v1
• Date: Wed, 02 Sep 2026 02:04:33 GMT
• Title: A Specialized Large Multimodal Model for Interpreting PET/CT in Head and Neck Cancer
• Abstract summary: Generalist Large Multimodal Models (LMMs) remain limited in medical contexts by insufficient domain-specific knowledge, privacy and security concerns, and verbosity.<n>We evaluated the feasibility of a specialized LMM for automated PET/CT interpretation in head and neck cancer.
• Score: 6.469240961927866
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: Diagnosing head and neck cancer using PET/CT is clinically challenging and time-consuming due to the anatomical complexity of the region, motivating computer-aided diagnosis (CAD). Generalist Large Multimodal Models (LMMs) remain limited in medical contexts by insufficient domain-specific knowledge, privacy and security concerns, and verbosity, motivating specialized standalone LMMs. Purpose: We evaluated the feasibility of a specialized LMM for automated PET/CT interpretation in head and neck cancer using a large-scale multi-institutional PET/CT dataset, a tailored training curriculum, and autoregressive training. Methods: LLaVA-NeXT was fine-tuned using a two-level curriculum with image-conversation pairs curated by two radiologists from public data. The dataset included clinically important annotations such as primary tumor presence and metastatic lymph node location. Level 1 used 28,000 image-conversation pairs to learn basic information, including modality type and hypermetabolism. Level 2 used 12,975 pairs to learn primary tumor presence and the existence and anatomical location of cervical lymph node metastases. External validation included four institutions with diverse imaging devices. Results: The specialized LMM substantially outperformed ChatGPT and LLaVA-NeXT. In Level-2 external validation, ROUGE-L, ROUGE-S, Cosine Similarity, Precision, Recall, and F1 were 0.8751, 0.8794, 0.8324, 0.8794, 0.8711, and 0.8751, while generalist models consistently scored below 0.1. Primary tumor classification accuracy was 83.14 +/- 1.15% internally and 69.03 +/- 0.81% externally. For lymph node localization, the corresponding scores were 0.6389, 0.6257, 0.5287, 0.5782, 0.6371, and 0.6648. Conclusion: Specialized LMMs show promising results for fast, accurate PET/CT-based diagnostic support and medical education, highlighting their potential for clinical translation.

Related papers

• A Vision-Language Foundation Model for Precise and Comprehensive Brain Tumor Diagnosis from Preoperative Multimodal Data [50.52606847963128]
  We developed BrainVLM to classify all 12 World Health Organization (WHO) 2021 brain tumor types.<n>BrainVLM was trained on multi-modal data (MRI scans, demographics, and radiology reports) from 40,043 individuals.<n>It was validated on 5,211 patients with pathologically confirmed brain tumors.
  arXiv  Detail & Related papers  (2026-09-15T03:46:03Z)
• Squeezed-Eff-Net: Edge-Computed Boost of Tomography Based Brain Tumor Classification leveraging Hybrid Neural Network Architecture [0.7829352305480285]
  This work proposes a hybrid deep learning model based on SqueezeNet v1 which is a lightweight model, and EfficientNet-B0, which is a high-performing model.<n>The framework was trained and tested only on publicly available Nickparvar Brain Tumor MRI dataset.
  arXiv  Detail & Related papers  (2025-12-08T07:37:30Z)
• An Anatomy Aware Hybrid Deep Learning Framework for Lung Cancer Tumor Stage Classification [9.447852227817533]
  We propose a medically grounded hybrid pipeline that performs staging by explicitly measuring the tumor's size and distance properties.<n>Our method employs specialized encoder-decoder networks to precisely segment the lung and adjacent anatomy.<n>This novel framework has been evaluated on the Lung-PET-CT-Dx dataset, demonstrating superior performance compared to traditional deep learning models.
  arXiv  Detail & Related papers  (2025-11-24T18:01:47Z)
• Handcrafted vs. Deep Radiomics vs. Fusion vs. Deep Learning: A Comprehensive Review of Machine Learning -Based Cancer Outcome Prediction in PET and SPECT Imaging [2.3507313809321233]
  This systematic review analyzed 226 studies published from 2020 to 2025 that applied machine learning to PET or SPECT imaging for outcome prediction.<n> PET-based studies generally outperformed those using SPECT, likely due to higher spatial resolution and sensitivity.<n>Common limitations included inadequate handling of class imbalance, missing data, and low population diversity.
  arXiv  Detail & Related papers  (2025-07-21T21:03:12Z)
• Predicting Length of Stay in Neurological ICU Patients Using Classical Machine Learning and Neural Network Models: A Benchmark Study on MIMIC-IV [49.1574468325115]
  This study explores multiple ML approaches for predicting LOS in ICU specifically for the patients with neurological diseases based on the MIMIC-IV dataset.<n>The evaluated models include classic ML algorithms (K-Nearest Neighbors, Random Forest, XGBoost and CatBoost) and Neural Networks (LSTM, BERT and Temporal Fusion Transformer)
  arXiv  Detail & Related papers  (2025-05-23T14:06:42Z)
• MAST-Pro: Dynamic Mixture-of-Experts for Adaptive Segmentation of Pan-Tumors with Knowledge-Driven Prompts [54.915060471994686]
  We propose MAST-Pro, a novel framework that integrates dynamic Mixture-of-Experts (D-MoE) and knowledge-driven prompts for pan-tumor segmentation.<n>Specifically, text and anatomical prompts provide domain-specific priors guiding tumor representation learning, while D-MoE dynamically selects experts to balance generic and tumor-specific feature learning.<n>Experiments on multi-anatomical tumor datasets demonstrate that MAST-Pro outperforms state-of-the-art approaches, achieving up to a 5.20% improvement in average improvement while reducing trainable parameters by 91.04%, without compromising accuracy.
  arXiv  Detail & Related papers  (2025-03-18T15:39:44Z)
• Analysis of the BraTS 2023 Intracranial Meningioma Segmentation Challenge [44.76736949127792]
  We describe the design and results from the BraTS 2023 Intracranial Meningioma Challenge.<n>The BraTS Meningioma Challenge differed from prior BraTS Glioma challenges in that it focused on meningiomas.<n>The top ranked team had a lesion-wise median dice similarity coefficient (DSC) of 0.976, 0.976, and 0.964 for enhancing tumor, tumor core, and whole tumor.
  arXiv  Detail & Related papers  (2024-05-16T03:23:57Z)
• 3D Lymphoma Segmentation on PET/CT Images via Multi-Scale Information Fusion with Cross-Attention [6.499725732124126]
  This study aims to develop a precise segmentation method for diffuse large B-cell lymphoma (DLBCL) lesions. We propose a 3D dual-branch encoder segmentation method using shifted window transformers and a Multi-Scale Information Fusion (MSIF) module. The model was trained and validated on a dataset of 165 DLBCL patients using 5-fold cross-validation.
  arXiv  Detail & Related papers  (2024-02-04T05:25:12Z)
• CancerUniT: Towards a Single Unified Model for Effective Detection, Segmentation, and Diagnosis of Eight Major Cancers Using a Large Collection of CT Scans [45.83431075462771]
  Human readers or radiologists routinely perform full-body multi-organ multi-disease detection and diagnosis in clinical practice. Most medical AI systems are built to focus on single organs with a narrow list of a few diseases. CancerUniT is a query-based Mask Transformer model with the output of multi-tumor prediction.
  arXiv  Detail & Related papers  (2023-01-28T20:09:34Z)
• ISA-Net: Improved spatial attention network for PET-CT tumor segmentation [22.48294544919023]
  We propose a deep learning segmentation method based on multimodal positron emission tomography-computed tomography (PET-CT) We design an improved spatial attention network(ISA-Net) to increase the accuracy of PET or CT in detecting tumors. We validated the proposed ISA-Net method on two clinical datasets, a soft tissue sarcoma(STS) and a head and neck tumor(HECKTOR) dataset.
  arXiv  Detail & Related papers  (2022-11-04T04:15:13Z)
• EMT-NET: Efficient multitask network for computer-aided diagnosis of breast cancer [58.720142291102135]
  We propose an efficient and light-weighted learning architecture to classify and segment breast tumors simultaneously. We incorporate a segmentation task into a tumor classification network, which makes the backbone network learn representations focused on tumor regions. The accuracy, sensitivity, and specificity of tumor classification is 88.6%, 94.1%, and 85.3%, respectively.
  arXiv  Detail & Related papers  (2022-01-13T05:24:40Z)
• Deep Learning-based Computational Pathology Predicts Origins for Cancers of Unknown Primary [2.645435564532842]
  Cancer of unknown primary (CUP) is an enigmatic group of diagnoses where the primary anatomical site of tumor origin cannot be determined. Recent work has focused on using genomics and transcriptomics for identification of tumor origins. We present a deep learning-based computational pathology algorithm that can provide a differential diagnosis for CUP.
  arXiv  Detail & Related papers  (2020-06-24T17:59:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.