Medical Image Understanding Improves Survival Prediction via Visual Instruction Tuning

• URL: http://arxiv.org/abs/2604.18250v1
• Date: Mon, 20 Apr 2026 13:27:39 GMT
• Title: Medical Image Understanding Improves Survival Prediction via Visual Instruction Tuning
• Abstract summary: We propose a vision-language framework for 3D CT image understanding that leverages large-scale open-sourced CT images and radiology reports.<n>Our approach improves survival prediction from CT images and clinical data while generating clinically meaningful language responses to predefined questions.
• Score: 8.133715585842355
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate prognostication and risk estimation are essential for guiding clinical decision-making and optimizing patient management. While radiologist-assessed features from CT scans provide valuable indicators of disease severity and outcomes, interpreting such images requires expert knowledge, and translating rich visual information into textual summaries inevitably leads to information loss. In this work, we propose a vision-language framework for 3D CT image understanding that leverages large-scale open-sourced CT images paired with radiology reports through visual instruction tuning. This pre-training enables the model to learn clinically meaningful visual-textual representations, which can then be adapted to downstream survival prediction tasks. By incorporating a survival prediction head on top of the pre-trained model, our approach improves survival prediction from CT images and clinical data while generating clinically meaningful language responses to predefined questions. Experimental results demonstrate that our method outperforms baseline methods in survival prediction, particularly, when clinical data alone is less predictive. The code will be released upon acceptance.

Related papers

• Dynamic Survival Prediction using Longitudinal Images based on Transformer [0.0]
  We introduce SurLonFormer, a novel Transformer-based neural network that integrates longitudinal medical imaging with structured data for survival prediction.<n>Our architecture comprises three key components: a Vision for extracting features, a Sequence for aggregating spatial temporal information, and a Survival based on the Cox proportional hazards model.
  arXiv  Detail & Related papers  (2025-08-12T20:31:55Z)
• Multi-task Learning Approach for Intracranial Hemorrhage Prognosis [0.0]
  We propose a 3D multi-task image model to predict prognosis, Glasgow Coma Scale and age, improving accuracy and interpretability. Our method outperforms current state-of-the-art baseline image models, and demonstrates superior performance in ICH prognosis compared to four board-certified neuroradiologists using only CT scans as input.
  arXiv  Detail & Related papers  (2024-08-16T14:56:17Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• Hypergraph Convolutional Networks for Fine-grained ICU Patient Similarity Analysis and Risk Prediction [15.06049250330114]
  The Intensive Care Unit (ICU) is one of the most important parts of a hospital, which admits critically ill patients and provides continuous monitoring and treatment. Various patient outcome prediction methods have been attempted to assist healthcare professionals in clinical decision-making.
  arXiv  Detail & Related papers  (2023-08-24T05:26:56Z)
• A Comparison of Self-Supervised Pretraining Approaches for Predicting Disease Risk from Chest Radiograph Images [3.5880535198436156]
  We present a comparison of semi- and self-supervised learning to predict mortality risk using chest x-ray images. We find that a semi-supervised autoencoder outperforms contrastive and transfer learning in internal and external validation.
  arXiv  Detail & Related papers  (2023-06-15T08:48:14Z)
• Vision-Language Modelling For Radiological Imaging and Reports In The Low Data Regime [70.04389979779195]
  This paper explores training medical vision-language models (VLMs) where the visual and language inputs are embedded into a common space. We explore several candidate methods to improve low-data performance, including adapting generic pre-trained models to novel image and text domains. Using text-to-image retrieval as a benchmark, we evaluate the performance of these methods with variable sized training datasets of paired chest X-rays and radiological reports.
  arXiv  Detail & Related papers  (2023-03-30T18:20:00Z)
• Temporal Context Matters: Enhancing Single Image Prediction with Disease Progression Representations [8.396615243014768]
  We present a deep learning approach that leverages temporal progression information to improve clinical outcome predictions from single-timepoint images. In our method, a self-attention based Temporal Convolutional Network (TCN) is used to learn a representation that is most reflective of the disease trajectory. A Vision Transformer is pretrained in a self-supervised fashion to extract features from single-timepoint images.
  arXiv  Detail & Related papers  (2022-03-02T22:11:07Z)
• Predicting Patient Readmission Risk from Medical Text via Knowledge Graph Enhanced Multiview Graph Convolution [67.72545656557858]
  We propose a new method that uses medical text of Electronic Health Records for prediction. We represent discharge summaries of patients with multiview graphs enhanced by an external knowledge graph. Experimental results prove the effectiveness of our method, yielding state-of-the-art performance.
  arXiv  Detail & Related papers  (2021-12-19T01:45:57Z)
• Variational Knowledge Distillation for Disease Classification in Chest X-Rays [102.04931207504173]
  We propose itvariational knowledge distillation (VKD), which is a new probabilistic inference framework for disease classification based on X-rays. We demonstrate the effectiveness of our method on three public benchmark datasets with paired X-ray images and EHRs.
  arXiv  Detail & Related papers  (2021-03-19T14:13:56Z)
• Deep Co-Attention Network for Multi-View Subspace Learning [73.3450258002607]
  We propose a deep co-attention network for multi-view subspace learning. It aims to extract both the common information and the complementary information in an adversarial setting. In particular, it uses a novel cross reconstruction loss and leverages the label information to guide the construction of the latent representation.
  arXiv  Detail & Related papers  (2021-02-15T18:46:44Z)
• Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration [55.88616573143478]
  Outcome prediction from clinical text can prevent doctors from overlooking possible risks. Diagnoses at discharge, procedures performed, in-hospital mortality and length-of-stay prediction are four common outcome prediction targets. We propose clinical outcome pre-training to integrate knowledge about patient outcomes from multiple public sources.
  arXiv  Detail & Related papers  (2021-02-08T10:26:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.