CXR-TFT: Multi-Modal Temporal Fusion Transformer for Predicting Chest X-ray Trajectories

• URL: http://arxiv.org/abs/2507.14766v1
• Date: Sat, 19 Jul 2025 22:42:26 GMT
• Title: CXR-TFT: Multi-Modal Temporal Fusion Transformer for Predicting Chest X-ray Trajectories
• Authors: Mehak Arora, Ayman Ali, Kaiyuan Wu, Carolyn Davis, Takashi Shimazui, Mahmoud Alwakeel, Victor Moas, Philip Yang, Annette Esper, Rishikesan Kamaleswaran,
• Abstract summary: In intensive care units (ICUs), patients with complex clinical conditions require vigilant monitoring and prompt interventions.<n>Existing tools for CXR interpretation are constrained by cross-sectional analysis, failing to capture temporal dynamics.<n>We introduce CXR-TFT, a novel multi-modal framework that integrates temporally sparse CXR imaging and radiology reports with high-frequency clinical data.
• Score: 1.681259205454531
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In intensive care units (ICUs), patients with complex clinical conditions require vigilant monitoring and prompt interventions. Chest X-rays (CXRs) are a vital diagnostic tool, providing insights into clinical trajectories, but their irregular acquisition limits their utility. Existing tools for CXR interpretation are constrained by cross-sectional analysis, failing to capture temporal dynamics. To address this, we introduce CXR-TFT, a novel multi-modal framework that integrates temporally sparse CXR imaging and radiology reports with high-frequency clinical data, such as vital signs, laboratory values, and respiratory flow sheets, to predict the trajectory of CXR findings in critically ill patients. CXR-TFT leverages latent embeddings from a vision encoder that are temporally aligned with hourly clinical data through interpolation. A transformer model is then trained to predict CXR embeddings at each hour, conditioned on previous embeddings and clinical measurements. In a retrospective study of 20,000 ICU patients, CXR-TFT demonstrated high accuracy in forecasting abnormal CXR findings up to 12 hours before they became radiographically evident. This predictive capability in clinical data holds significant potential for enhancing the management of time-sensitive conditions like acute respiratory distress syndrome, where early intervention is crucial and diagnoses are often delayed. By providing distinctive temporal resolution in prognostic CXR analysis, CXR-TFT offers actionable 'whole patient' insights that can directly improve clinical outcomes.

Related papers

• CXR-LT 2024: A MICCAI challenge on long-tailed, multi-label, and zero-shot disease classification from chest X-ray [64.2434525370243]
  The CXR-LT series is a community-driven initiative designed to enhance lung disease classification using chest X-rays.<n>The CXR-LT 2024 expands the dataset to 377,110 chest X-rays (CXRs) and 45 disease labels, including 19 new rare disease findings.<n>This paper provides an overview of CXR-LT 2024, detailing the data curation process and consolidating state-of-the-art solutions.
  arXiv  Detail & Related papers  (2025-06-09T17:53:31Z)
• Addressing Asynchronicity in Clinical Multimodal Fusion via Individualized Chest X-ray Generation [14.658627367126009]
  We propose DDL-CXR, a method that dynamically generates an up-to-date latent representation of the individualized chest X-ray images. Our approach leverages latent diffusion models for patient-specific generation strategically conditioned on a previous CXR image and EHR time series. Experiments using MIMIC datasets show that the proposed model could effectively address asynchronicity in multimodal fusion and consistently outperform existing methods.
  arXiv  Detail & Related papers  (2024-10-23T14:34:39Z)
• Towards Predicting Temporal Changes in a Patient's Chest X-ray Images based on Electronic Health Records [9.398163873685798]
  We propose a novel framework, EHRXDiff, which predicts future CXR images by integrating previous CXRs with subsequent medical events.<n>Results show that our framework generates high-quality, realistic future images that effectively capture potential temporal changes.
  arXiv  Detail & Related papers  (2024-09-11T04:49:44Z)
• Expert Uncertainty and Severity Aware Chest X-Ray Classification by Multi-Relationship Graph Learning [48.29204631769816]
  We re-extract disease labels from CXR reports to make them more realistic by considering disease severity and uncertainty in classification. Our experimental results show that models considering disease severity and uncertainty outperform previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-09-06T19:19:41Z)
• Longitudinal Data and a Semantic Similarity Reward for Chest X-Ray Report Generation [7.586632627817609]
  Radiologists face high burnout rates, partly due to the increasing volume of Chest X-rays (CXRs) requiring interpretation and reporting. Our proposed CXR report generator integrates elements of the workflow and introduces a novel reward for reinforcement learning. Results from our study demonstrate that the proposed model generates reports that are more aligned with radiologists' reports than state-of-the-art models.
  arXiv  Detail & Related papers  (2023-07-19T05:41:14Z)
• Revisiting Computer-Aided Tuberculosis Diagnosis [56.80999479735375]
  Tuberculosis (TB) is a major global health threat, causing millions of deaths annually. Computer-aided tuberculosis diagnosis (CTD) using deep learning has shown promise, but progress is hindered by limited training data. We establish a large-scale dataset, namely the Tuberculosis X-ray (TBX11K) dataset, which contains 11,200 chest X-ray (CXR) images with corresponding bounding box annotations for TB areas. This dataset enables the training of sophisticated detectors for high-quality CTD.
  arXiv  Detail & Related papers  (2023-07-06T08:27:48Z)
• Optimising Chest X-Rays for Image Analysis by Identifying and Removing Confounding Factors [49.005337470305584]
  During the COVID-19 pandemic, the sheer volume of imaging performed in an emergency setting for COVID-19 diagnosis has resulted in a wide variability of clinical CXR acquisitions. The variable quality of clinically-acquired CXRs within publicly available datasets could have a profound effect on algorithm performance. We propose a simple and effective step-wise approach to pre-processing a COVID-19 chest X-ray dataset to remove undesired biases.
  arXiv  Detail & Related papers  (2022-08-22T13:57:04Z)
• COVID-Net CXR-S: Deep Convolutional Neural Network for Severity Assessment of COVID-19 Cases from Chest X-ray Images [74.77272804752306]
  We introduce COVID-Net CXR-S, a convolutional neural network for predicting the airspace severity of a SARS-CoV-2 positive patient based on a CXR image of the patient's chest. We leveraged transfer learning to transfer representational knowledge gained from over 16,000 CXR images from a multinational cohort of over 15,000 patient cases into a custom network architecture for severity assessment. The proposed COVID-Net CXR-S has potential to be a powerful tool for computer-aided severity assessment of CXR images of COVID-19 positive patients.
  arXiv  Detail & Related papers  (2021-05-01T14:15:12Z)
• AIforCOVID: predicting the clinical outcomes in patients with COVID-19 applying AI to chest-X-rays. An Italian multicentre study [7.456548336226919]
  We investigate whether chest X-ray (CXR) can be used as a possible tool for the early identification of patients at risk of severe outcome, like intensive care or death. CXR is a radiological technique that compared to computed tomography (CT) it is simpler, faster, more widespread and it induces lower radiation dose. We present a dataset including data collected from 820 patients by six Italian hospitals in spring 2020 during the first COVID-19 emergency.
  arXiv  Detail & Related papers  (2020-12-11T18:03:08Z)
• MIA-Prognosis: A Deep Learning Framework to Predict Therapy Response [58.0291320452122]
  This paper aims at a unified deep learning approach to predict patient prognosis and therapy response. We formalize the prognosis modeling as a multi-modal asynchronous time series classification task. Our predictive model could further stratify low-risk and high-risk patients in terms of long-term survival.
  arXiv  Detail & Related papers  (2020-10-08T15:30:17Z)
• Integrative Analysis for COVID-19 Patient Outcome Prediction [53.11258640541513]
  We combine radiomics of lung opacities and non-imaging features from demographic data, vital signs, and laboratory findings to predict need for intensive care unit admission. Our methods may also be applied to other lung diseases including but not limited to community acquired pneumonia.
  arXiv  Detail & Related papers  (2020-07-20T19:08:50Z)
• AI-Driven CT-based quantification, staging and short-term outcome prediction of COVID-19 pneumonia [3.672093204122992]
  Chest computed tomography (CT) is widely used for the management of Coronavirus disease 2019 (COVID-19) pneumonia. CT has a prognostic role by allowing visually evaluating the extent of COVID-19 lung abnormalities.
  arXiv  Detail & Related papers  (2020-04-20T12:24:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.