Comprehensive Multimodal Deep Learning Survival Prediction Enabled by a Transformer Architecture: A Multicenter Study in Glioblastoma

• URL: http://arxiv.org/abs/2405.12963v1
• Date: Tue, 21 May 2024 17:44:48 GMT
• Title: Comprehensive Multimodal Deep Learning Survival Prediction Enabled by a Transformer Architecture: A Multicenter Study in Glioblastoma
• Abstract summary: This research aims to improve glioblastoma survival prediction by integrating MR images, clinical and molecular-pathologic data in a transformer-based deep learning model. The model employs self-supervised learning techniques to effectively encode the high-dimensional MRI input for integration with non-imaging data using cross-attention.
• Score: 4.578027879885667
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Background: This research aims to improve glioblastoma survival prediction by integrating MR images, clinical and molecular-pathologic data in a transformer-based deep learning model, addressing data heterogeneity and performance generalizability. Method: We propose and evaluate a transformer-based non-linear and non-proportional survival prediction model. The model employs self-supervised learning techniques to effectively encode the high-dimensional MRI input for integration with non-imaging data using cross-attention. To demonstrate model generalizability, the model is assessed with the time-dependent concordance index (Cdt) in two training setups using three independent public test sets: UPenn-GBM, UCSF-PDGM, and RHUH-GBM, each comprising 378, 366, and 36 cases, respectively. Results: The proposed transformer model achieved promising performance for imaging as well as non-imaging data, effectively integrating both modalities for enhanced performance (UPenn-GBM test-set, imaging Cdt 0.645, multimodal Cdt 0.707) while outperforming state-of-the-art late-fusion 3D-CNN-based models. Consistent performance was observed across the three independent multicenter test sets with Cdt values of 0.707 (UPenn-GBM, internal test set), 0.672 (UCSF-PDGM, first external test set) and 0.618 (RHUH-GBM, second external test set). The model achieved significant discrimination between patients with favorable and unfavorable survival for all three datasets (logrank p 1.9\times{10}^{-8}, 9.7\times{10}^{-3}, and 1.2\times{10}^{-2}). Conclusions: The proposed transformer-based survival prediction model integrates complementary information from diverse input modalities, contributing to improved glioblastoma survival prediction compared to state-of-the-art methods. Consistent performance was observed across institutions supporting model generalizability.

Related papers

• Multi-cancer detection using a computationally efficient CNN with transfer learning [0.4889686357485836]
  This study introduces a computationally efficient convolutional neural network (CNN) architecture enhanced with transfer learning for multi-cancer detection.<n>We evaluate this approach on three distinct tumor datasets comprising brain magnetic resonance imaging (MRI) and lung and kidney computed tomography (CT) scans.<n>The model achieves test accuracy of 90.85 +- 2.22%, 98.64 +- 2.43% and 99.92 +- 0.08% for brain, lung, and kidney cancer classification, respectively.
  arXiv  Detail & Related papers  (2026-06-21T09:12:44Z)
• D3Seg: Dependency-Aware Diffusion for Brain Tumor Segmentation with Missing Modalities [54.87947751720332]
  We propose a novel segmentation model D3Seg to maintain stable performance under missing-modality settings.<n>The proposed model achieves approximately 1.5-2.0% Dice improvement on enhancing tumor (ET) and around 1.0% on tumor core (TC) across multiple missing modality configurations.
  arXiv  Detail & Related papers  (2026-05-21T09:55:42Z)
• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• A Heterogeneous Ensemble for Multi-Center COVID-19 Classification from Chest CT Scans [0.39998518782208775]
  We present a heterogeneous ensemble of nine models spanning three inference paradigms.<n>The ensemble achieves an average macro F1 of 0.9280 across four hospital centres, outperforming the best single model.
  arXiv  Detail & Related papers  (2026-03-15T21:34:59Z)
• Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attribution [42.85462513661566]
  We present a lightweight, transparent multimodal ensemble that fuses physiological time-series measurements with unstructured clinical notes from the first 48 hours of an ICU stay.<n>A logistic regression model combines predictions from two modality-specific models: a bidirectional LSTM for vitals and a finetuned ClinicalModernBERT transformer for notes.<n>On the MIMIC-III benchmark, our late-fusion ensemble improves discrimination over the best single model while maintaining well-calibrated predictions.
  arXiv  Detail & Related papers  (2025-11-19T20:11:49Z)
• Cross-dataset Multivariate Time-series Model for Parkinson's Diagnosis via Keyboard Dynamics [1.4895002251586484]
  Parkinson's disease (PD) presents a growing global challenge, affecting over 10 million individuals.<n>We propose a novel pipeline that leverages keystroke dynamics as a non-invasive and scalable biomarker for remote PD screening and telemonitoring.
  arXiv  Detail & Related papers  (2025-10-10T17:42:08Z)
• impuTMAE: Multi-modal Transformer with Masked Pre-training for Missing Modalities Imputation in Cancer Survival Prediction [75.43342771863837]
  We introduce impuTMAE, a novel transformer-based end-to-end approach with an efficient multimodal pre-training strategy.<n>It learns inter- and intra-modal interactions while simultaneously imputing missing modalities by reconstructing masked patches.<n>Our model is pre-trained on heterogeneous, incomplete data and fine-tuned for glioma survival prediction using TCGA-GBM/LGG and BraTS datasets.
  arXiv  Detail & Related papers  (2025-08-08T10:01:16Z)
• Predicting Length of Stay in Neurological ICU Patients Using Classical Machine Learning and Neural Network Models: A Benchmark Study on MIMIC-IV [49.1574468325115]
  This study explores multiple ML approaches for predicting LOS in ICU specifically for the patients with neurological diseases based on the MIMIC-IV dataset.<n>The evaluated models include classic ML algorithms (K-Nearest Neighbors, Random Forest, XGBoost and CatBoost) and Neural Networks (LSTM, BERT and Temporal Fusion Transformer)
  arXiv  Detail & Related papers  (2025-05-23T14:06:42Z)
• A multimodal ensemble approach for clear cell renal cell carcinoma treatment outcome prediction [6.199310532720352]
  We developed a multi-modal ensemble model (MMEM) that integrates clinical data, multi-omics data, and histopathology whole slide image (WSI) data. MMEM predicted overall survival (OS) and disease-free survival (DFS) for ccRCC patients.
  arXiv  Detail & Related papers  (2024-12-10T02:51:14Z)
• CRTRE: Causal Rule Generation with Target Trial Emulation Framework [47.2836994469923]
  We introduce a novel method called causal rule generation with target trial emulation framework (CRTRE) CRTRE applies randomize trial design principles to estimate the causal effect of association rules. We then incorporate such association rules for the downstream applications such as prediction of disease onsets.
  arXiv  Detail & Related papers  (2024-11-10T02:40:06Z)
• 2D and 3D Deep Learning Models for MRI-based Parkinson's Disease Classification: A Comparative Analysis of Convolutional Kolmogorov-Arnold Networks, Convolutional Neural Networks, and Graph Convolutional Networks [0.0]
  This study applies Convolutional Kolmogorov-Arnold Networks (ConvKANs) to Parkinson's Disease diagnosis. ConvKANs integrate learnable activation functions into convolutional layers, for PD classification using structural MRI. The first 3D implementation of ConvKANs for medical imaging is presented, comparing their performance to Convolutional Neural Networks (CNNs) and Graph Convolutional Networks (GCNs) These findings highlight ConvKANs' potential for PD detection, emphasize the importance of 3D analysis in capturing subtle brain changes, and underscore cross-dataset generalization challenges.
  arXiv  Detail & Related papers  (2024-07-24T16:04:18Z)
• Brain Tumor Radiogenomic Classification [1.8276368987462532]
  The RSNA-MICCAI brain tumor radiogenomic classification challenge aimed to predict MGMT biomarker status in glioblastoma through binary classification. The dataset is splitted into three main cohorts: training set, validation set which were used during training, and the testing were only used during final evaluation.
  arXiv  Detail & Related papers  (2024-01-11T10:30:09Z)
• The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection [51.697248252191265]
  This work summarizes and strictly observes best practices regarding data handling, experimental design, and model evaluation. We focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. Within this framework, we train predictive 15 models, considering three different data augmentation strategies and five distinct 3D CNN architectures.
  arXiv  Detail & Related papers  (2023-09-13T10:40:41Z)
• Virtual imaging trials improved the transparency and reliability of AI systems in COVID-19 imaging [1.6040478776985583]
  This study focuses on using convolutional neural networks (CNNs) for COVID-19 diagnosis using computed tomography (CT) and chest radiography (CXR) We developed and tested multiple AI models, 3D ResNet-like and 2D EfficientNetv2 architectures, across diverse datasets. Models trained on the most diverse datasets showed the highest external testing performance, with AUC values ranging from 0.73 to 0.76 for CT and 0.70 to 0.73 for CXR.
  arXiv  Detail & Related papers  (2023-08-17T19:12:32Z)
• Breast Ultrasound Tumor Classification Using a Hybrid Multitask CNN-Transformer Network [63.845552349914186]
  Capturing global contextual information plays a critical role in breast ultrasound (BUS) image classification. Vision Transformers have an improved capability of capturing global contextual information but may distort the local image patterns due to the tokenization operations. In this study, we proposed a hybrid multitask deep neural network called Hybrid-MT-ESTAN, designed to perform BUS tumor classification and segmentation.
  arXiv  Detail & Related papers  (2023-08-04T01:19:32Z)
• Brain Imaging-to-Graph Generation using Adversarial Hierarchical Diffusion Models for MCI Causality Analysis [44.45598796591008]
  Brain imaging-to-graph generation (BIGG) framework is proposed to map functional magnetic resonance imaging (fMRI) into effective connectivity for mild cognitive impairment analysis. The hierarchical transformers in the generator are designed to estimate the noise at multiple scales. Evaluations of the ADNI dataset demonstrate the feasibility and efficacy of the proposed model.
  arXiv  Detail & Related papers  (2023-05-18T06:54:56Z)
• On the explainability of hospitalization prediction on a large COVID-19 patient dataset [45.82374977939355]
  We develop various AI models to predict hospitalization on a large (over 110$k$) cohort of COVID-19 positive-tested US patients. Despite high data unbalance, the models reach average precision 0.96-0.98 (0.75-0.85), recall 0.96-0.98 (0.74-0.85), and $F_score 0.97-0.98 (0.79-0.83) on the non-hospitalized (or hospitalized) class.
  arXiv  Detail & Related papers  (2021-10-28T10:23:38Z)
• Vision Transformers for femur fracture classification [59.99241204074268]
  The Vision Transformer (ViT) was able to correctly predict 83% of the test images. Good results were obtained in sub-fractures with the largest and richest dataset ever.
  arXiv  Detail & Related papers  (2021-08-07T10:12:42Z)
• A multi-stage machine learning model on diagnosis of esophageal manometry [50.591267188664666]
  The framework includes deep-learning models at the swallow-level stage and feature-based machine learning models at the study-level stage. This is the first artificial-intelligence-style model to automatically predict CC diagnosis of HRM study from raw multi-swallow data.
  arXiv  Detail & Related papers  (2021-06-25T20:09:23Z)
• Bootstrapping Your Own Positive Sample: Contrastive Learning With Electronic Health Record Data [62.29031007761901]
  This paper proposes a novel contrastive regularized clinical classification model. We introduce two unique positive sampling strategies specifically tailored for EHR data. Our framework yields highly competitive experimental results in predicting the mortality risk on real-world COVID-19 EHR data.
  arXiv  Detail & Related papers  (2021-04-07T06:02:04Z)
• Deep learning-based COVID-19 pneumonia classification using chest CT images: model generalizability [54.86482395312936]
  Deep learning (DL) classification models were trained to identify COVID-19-positive patients on 3D computed tomography (CT) datasets from different countries. We trained nine identical DL-based classification models by using combinations of the datasets with a 72% train, 8% validation, and 20% test data split. The models trained on multiple datasets and evaluated on a test set from one of the datasets used for training performed better.
  arXiv  Detail & Related papers  (2021-02-18T21:14:52Z)
• Interpretable Machine Learning Model for Early Prediction of Mortality in Elderly Patients with Multiple Organ Dysfunction Syndrome (MODS): a Multicenter Retrospective Study and Cross Validation [9.808639780672156]
  Elderly patients with MODS have high risk of death and poor prognosis. This study aims to develop an interpretable and generalizable model for early mortality prediction in elderly patients with MODS.
  arXiv  Detail & Related papers  (2020-01-28T17:15:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.