Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk Assessment

• URL: http://arxiv.org/abs/2606.22442v1
• Date: Sun, 21 Jun 2026 11:18:06 GMT
• Title: Efficient Multimodal Clinical Question Answering for Pulmonary Embolism Risk Assessment
• Abstract summary: Pulmonary embolism (PE) is a high risk cardiopulmonary condition whose management requires timely diagnosis and reliable assessment of future clinical risk.<n>Because PE care routinely combines computed tomography pulmonary angiography (CTPA), radiology interpretation, and longitudinal electronic health record (EHR) evidence, it provides a clinically meaningful setting for evaluating compact multimodal language models.<n>In this work, we build a benchmark using efficient multimodal large language models (MLLMs) on INSPECT, a multimodal PE dataset containing 23,248A studies from 19,402 patients.
• Score: 18.646810467653484
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pulmonary embolism (PE) is a high risk cardiopulmonary condition whose management requires both timely diagnosis and reliable assessment of future clinical risk. Because PE care routinely combines computed tomography pulmonary angiography (CTPA), radiology interpretation, and longitudinal electronic health record (EHR) evidence, it provides a clinically meaningful setting for evaluating compact multimodal language models. In this work, we build a benchmark using efficient multimodal large language models (MLLMs) on INSPECT, a multimodal PE dataset containing 23,248 CTPA studies from 19,402 patients. We formulate eight diagnostic and prognostic tasks as structured clinical question answering problems and evaluate on typical efficient MLLMs under CTPA-Only, EHR-Only, and CTPA+EHR settings with zero-shot and few-shot prompting. Results show that Gemma4 E4B and Gemma4 E2B perform more strongly when EHR evidence is available, especially under CTPA+EHR input. Task level analysis further shows that PE diagnosis achieves higher performance than prognostic tasks, particularly readmission prediction. These observations suggest that compact multimodal models have the great potential in early stage PE risk detection and explanation.

Related papers

• Prospective evaluation of multimodal respiratory failure prediction: Do chest X-rays improve performance beyond EHR signals? [4.250825424649631]
  We develop a gated multimodal framework that integrates structured EHR time-series data with CXR foundation-model representations.<n>We evaluate the framework for predicting invasive mechanical ventilation within 24 hours in ICU patients.
  arXiv  Detail & Related papers  (2026-05-25T18:25:47Z)
• EviCare: Enhancing Diagnosis Prediction with Deep Model-Guided Evidence for In-Context Reasoning [62.61394722212386]
  EviCare is an in-context reasoning framework that integrates deep model guidance into large language models.<n>It consistently outperforms both LLM-only and deep model-only baselines in experiments on two real-world EHR benchmarks.
  arXiv  Detail & Related papers  (2026-04-12T04:35:14Z)
• TumorChain: Interleaved Multimodal Chain-of-Thought Reasoning for Traceable Clinical Tumor Analysis [46.04720262017957]
  TumorChain is a multimodal interleaved reasoning framework that tightly couples 3D imaging encoders, clinical text understanding, and organ-level vision-language alignment.<n>Experiments show consistent improvements over strong baselines in lesion detection, impression generation, and pathology classification, and demonstrate strong generalization on the DeepTumorVQA benchmark.
  arXiv  Detail & Related papers  (2026-03-06T03:42:22Z)
• UniPACT: A Multimodal Framework for Prognostic Question Answering on Raw ECG and Structured EHR [15.533178197005208]
  We propose a unified framework for prognostic question answering that bridges this modality gap.<n>UniPACT's core contribution is a structured prompting mechanism that converts numerical EHR data into semantically rich text.<n>We evaluate it on the comprehensive MDS-ED benchmark, it achieves a state-of-the-art mean AUROC of 89.37% across a diverse set of prognostic tasks.
  arXiv  Detail & Related papers  (2026-01-25T17:35:52Z)
• EHR-R1: A Reasoning-Enhanced Foundational Language Model for Electronic Health Record Analysis [78.05962117947259]
  We present EHR-Ins, a large-scale, comprehensive EHR reasoning instruction dataset.<n>We also develop EHR-R1, a series of reasoning-enhanced LLMs with up to 72B parameters tailored for EHR analysis.<n>Finally, we introduce EHR-Bench, a new benchmark curated from MIMIC-IV, spanning 42 tasks.
  arXiv  Detail & Related papers  (2025-10-29T15:32:47Z)
• LlaMADRS: Prompting Large Language Models for Interview-Based Depression Assessment [75.44934940580112]
  This study introduces LlaMADRS, a novel framework leveraging open-source Large Language Models (LLMs) to automate depression severity assessment.<n>We employ a zero-shot prompting strategy with carefully designed cues to guide the model in interpreting and scoring transcribed clinical interviews.<n>Our approach, tested on 236 real-world interviews, demonstrates strong correlations with clinician assessments.
  arXiv  Detail & Related papers  (2025-01-07T08:49:04Z)
• CTPD: Cross-Modal Temporal Pattern Discovery for Enhanced Multimodal Electronic Health Records Analysis [50.56875995511431]
  We introduce a Cross-Modal Temporal Pattern Discovery (CTPD) framework, designed to efficiently extract meaningful cross-modal temporal patterns from multimodal EHR data.<n>Our approach introduces shared initial temporal pattern representations which are refined using slot attention to generate temporal semantic embeddings.
  arXiv  Detail & Related papers  (2024-11-01T15:54:07Z)
• Detecting and clustering swallow events in esophageal long-term high-resolution manometry [48.688209040613216]
  We propose a Deep Learning based swallowing detection method to accurately identify swallowing events and secondary non-deglutitive-induced esophageal motility disorders. We evaluate our computational pipeline on a total of 25 LTHRMs, which were meticulously annotated by medical experts.
  arXiv  Detail & Related papers  (2024-05-02T09:41:31Z)
• INSPECT: A Multimodal Dataset for Pulmonary Embolism Diagnosis and Prognosis [19.32686665459374]
  We introduce INSPECT, which contains de-identified longitudinal records from a large cohort of patients at risk for pulmonary embolism (PE) INSPECT contains data from 19,402 patients, including CT images, radiology report impression sections, and structured electronic health record (EHR) data (i.e. demographics, diagnoses, procedures, vitals, and medications)
  arXiv  Detail & Related papers  (2023-11-17T07:28:16Z)
• Towards Reliable and Explainable AI Model for Solid Pulmonary Nodule Diagnosis [20.510918720980467]
  Lung cancer has the highest mortality rate of deadly cancers in the world. Computer-aided diagnosis (CAD) systems have been developed to assist radiologists in nodule detection and diagnosis. Lack of model reliability and interpretability remains a major obstacle for its large-scale clinical application.
  arXiv  Detail & Related papers  (2022-04-08T08:21:00Z)
• Attention based CNN-LSTM Network for Pulmonary Embolism Prediction on Chest Computed Tomography Pulmonary Angiograms [22.62583095023903]
  Pulmonary Embolism (PE) is among the most fatal cardiovascular diseases. We propose a two-stage attention-based CNN-LSTM network for predicting PE. Our framework mirrors the radiologic diagnostic process via a multi-slice approach.
  arXiv  Detail & Related papers  (2021-07-13T17:58:15Z)
• Hemogram Data as a Tool for Decision-making in COVID-19 Management: Applications to Resource Scarcity Scenarios [62.997667081978825]
  COVID-19 pandemics has challenged emergency response systems worldwide, with widespread reports of essential services breakdown and collapse of health care structure. This work describes a machine learning model derived from hemogram exam data performed in symptomatic patients. Proposed models can predict COVID-19 qRT-PCR results in symptomatic individuals with high accuracy, sensitivity and specificity.
  arXiv  Detail & Related papers  (2020-05-10T01:45:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.