Multimodal Deep Learning for Early Prediction of Patient Deterioration in the ICU: Integrating Time-Series EHR Data with Clinical Notes

• URL: http://arxiv.org/abs/2603.14719v1
• Date: Mon, 16 Mar 2026 01:50:08 GMT
• Title: Multimodal Deep Learning for Early Prediction of Patient Deterioration in the ICU: Integrating Time-Series EHR Data with Clinical Notes
• Abstract summary: Early identification of patients at risk for clinical deterioration in the intensive care unit (ICU) remains a critical challenge.<n>We present a multimodal deep learning approach that combines structured time-series data (vital signs and laboratory values) with unstructured clinical notes to predict patient deterioration within 24 hours.<n>Our architecture employs a bidirectional LSTM encoder for temporal patterns in physiologic data and ClinicalBERT embeddings for clinical notes, fused through a cross-modal attention mechanism.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Early identification of patients at risk for clinical deterioration in the intensive care unit (ICU) remains a critical challenge. Delayed recognition of impending adverse events, including mortality, vasopressor initiation, and mechanical ventilation, contributes to preventable morbidity and mortality. We present a multimodal deep learning approach that combines structured time-series data (vital signs and laboratory values) with unstructured clinical notes to predict patient deterioration within 24 hours. Using the MIMIC-IV database, we constructed a cohort of 74,822 ICU stays and generated 5.7 million hourly prediction samples. Our architecture employs a bidirectional LSTM encoder for temporal patterns in physiologic data and ClinicalBERT embeddings for clinical notes, fused through a cross-modal attention mechanism. We also present a systematic review of existing approaches to ICU deterioration prediction, identifying 31 studies published between 2015 and 2024. Most existing models rely solely on structured data and achieve area under the curve (AUC) values between 0.70 and 0.85. Studies incorporating clinical notes remain rare but show promise for capturing information not present in structured fields. Our multimodal model achieves a test AUROC of 0.7857 and AUPRC of 0.1908 on 823,641 held-out samples, with a validation-to-test gap of only 0.6 percentage points. Ablation analysis validates the multimodal approach: clinical notes improve AUROC by 2.5 percentage points and AUPRC by 39.2% relative to a structured-only baseline, while deep learning models consistently outperform classical baselines (XGBoost AUROC: 0.7486, logistic regression: 0.7171). This work contributes both a thorough review of the field and a reproducible multimodal framework for clinical deterioration prediction.

Related papers

• Cerebra: A Multidisciplinary AI Board for Multimodal Dementia Characterization and Risk Assessment [56.62016795093786]
  Cerebra is an interactive multi-agent AI team that coordinates specialized agents for EHR, clinical notes, and medical imaging analysis.<n>It supports privacy-preserving deployment by operating on structured representations and remains robust when modalities are incomplete.<n>Cerebra significantly improved expert performance, increasing accuracy by 17.5 percentage points in prospective dementia risk estimation.
  arXiv  Detail & Related papers  (2026-03-23T05:46:45Z)
• A Multimodal Deep Learning Framework for Predicting ICU Deterioration: Integrating ECG Waveforms with Clinical Data and Clinician Benchmarking [1.2938481923855734]
  MDS ICU is a unified machine learning framework that fuses routinely collected data including demographics, biometrics, vital signs, laboratory values, ECG waveforms, surgical procedures, and medical device usage to provide continuous predictive support during ICU stays.<n>We trained a deep learning architecture that combines structured state space S4 encoders for ECG waveforms with multilayer perceptron RealMLP encoders for tabular data to jointly predict 33 clinically relevant outcomes spanning mortality, organ dysfunction, medication needs, and acute deterioration.<n>Analysis showed close agreement between predicted and observed risks, with consistent gains from ECG waveform integration.
  arXiv  Detail & Related papers  (2026-01-10T18:11:12Z)
• Development and external validation of a multimodal artificial intelligence mortality prediction model of critically ill patients using multicenter data [1.3352724459394656]
  The objective was to develop a multimodal deep learning model, using structured and unstructured clinical data, to predict in-hospital mortality risk.<n>We used data from MIMIC-III, MIMIC-IV, eICU, and HiRID.<n>A total of 203,434 ICU admissions from more than 200 hospitals between 2001 to 2022 were included.
  arXiv  Detail & Related papers  (2025-12-15T23:43:19Z)
• Transparent Early ICU Mortality Prediction with Clinical Transformer and Per-Case Modality Attribution [42.85462513661566]
  We present a lightweight, transparent multimodal ensemble that fuses physiological time-series measurements with unstructured clinical notes from the first 48 hours of an ICU stay.<n>A logistic regression model combines predictions from two modality-specific models: a bidirectional LSTM for vitals and a finetuned ClinicalModernBERT transformer for notes.<n>On the MIMIC-III benchmark, our late-fusion ensemble improves discrimination over the best single model while maintaining well-calibrated predictions.
  arXiv  Detail & Related papers  (2025-11-19T20:11:49Z)
• Enhancing mortality prediction in cardiac arrest ICU patients through meta-modeling of structured clinical data from MIMIC-IV [0.0]
  This study develops and evaluates machine learning models that integrate structured clinical data and unstructured information.<n>We used LASSO and XGBoost for feature selection, followed by a logistic regression trained on the top features identified by both models.<n>The final logistic regression model, which combined structured and textual input, achieved an AUC of 0.918, compared to 0.753 when using structured data alone, a relative improvement 22%.
  arXiv  Detail & Related papers  (2025-10-20T20:56:45Z)
• A Disease-Centric Vision-Language Foundation Model for Precision Oncology in Kidney Cancer [54.58205672910646]
  RenalCLIP is a visual-language foundation model for characterization, diagnosis and prognosis of renal mass.<n>It achieved better performance and superior generalizability across 10 core tasks spanning the full clinical workflow of kidney cancer.
  arXiv  Detail & Related papers  (2025-08-22T17:48:19Z)
• Early Mortality Prediction in ICU Patients with Hypertensive Kidney Disease Using Interpretable Machine Learning [3.4335475695580127]
  Hypertensive kidney disease (HKD) patients in intensive care units (ICUs) face high short-term mortality.<n>We developed a machine learning framework to predict 30-day in-hospital mortality among ICU patients with HKD.
  arXiv  Detail & Related papers  (2025-07-25T00:48:23Z)
• Development of Interactive Nomograms for Predicting Short-Term Survival in ICU Patients with Aplastic Anemia [3.5626691568652507]
  Aplastic anemia is a rare, life-threatening hematologic disorder characterized by pancytopenia and bone marrow failure.<n>We used the MIMIC-IV database to identify ICU patients with aplastic anemia and extracted clinical features from five domains.<n> Logistic regression and Cox regression models were constructed to predict 7-, 14-, and 28-day mortality.
  arXiv  Detail & Related papers  (2025-05-23T23:01:11Z)
• Predicting Length of Stay in Neurological ICU Patients Using Classical Machine Learning and Neural Network Models: A Benchmark Study on MIMIC-IV [49.1574468325115]
  This study explores multiple ML approaches for predicting LOS in ICU specifically for the patients with neurological diseases based on the MIMIC-IV dataset.<n>The evaluated models include classic ML algorithms (K-Nearest Neighbors, Random Forest, XGBoost and CatBoost) and Neural Networks (LSTM, BERT and Temporal Fusion Transformer)
  arXiv  Detail & Related papers  (2025-05-23T14:06:42Z)
• Deep State-Space Generative Model For Correlated Time-to-Event Predictions [54.3637600983898]
  We propose a deep latent state-space generative model to capture the interactions among different types of correlated clinical events. Our method also uncovers meaningful insights about the latent correlations among mortality and different types of organ failures.
  arXiv  Detail & Related papers  (2024-07-28T02:42:36Z)
• Multimodal Pretraining of Medical Time Series and Notes [45.89025874396911]
  Deep learning models show promise in extracting meaningful patterns, but they require extensive labeled data. We propose a novel approach employing self-supervised pretraining, focusing on the alignment of clinical measurements and notes. In downstream tasks, including in-hospital mortality prediction and phenotyping, our model outperforms baselines in settings where only a fraction of the data is labeled.
  arXiv  Detail & Related papers  (2023-12-11T21:53:40Z)
• MIA-Prognosis: A Deep Learning Framework to Predict Therapy Response [58.0291320452122]
  This paper aims at a unified deep learning approach to predict patient prognosis and therapy response. We formalize the prognosis modeling as a multi-modal asynchronous time series classification task. Our predictive model could further stratify low-risk and high-risk patients in terms of long-term survival.
  arXiv  Detail & Related papers  (2020-10-08T15:30:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.