Medical Video Generation for Disease Progression Simulation

• URL: http://arxiv.org/abs/2411.11943v1
• Date: Mon, 18 Nov 2024 18:37:09 GMT
• Title: Medical Video Generation for Disease Progression Simulation
• Authors: Xu Cao, Kaizhao Liang, Kuei-Da Liao, Tianren Gao, Wenqian Ye, Jintai Chen, Zhiguang Ding, Jianguo Cao, James M. Rehg, Jimeng Sun,
• Abstract summary: We propose the first Medical Video Generation framework that enables controlled manipulation of disease-related image and video features. We validate our framework across three medical imaging domains: chest X-ray, fundus photography, and skin image. MVG significantly outperforms baseline models in generating coherent and clinically plausible disease trajectories.
• Score: 40.38123964910394
• License:
• Abstract: Modeling disease progression is crucial for improving the quality and efficacy of clinical diagnosis and prognosis, but it is often hindered by a lack of longitudinal medical image monitoring for individual patients. To address this challenge, we propose the first Medical Video Generation (MVG) framework that enables controlled manipulation of disease-related image and video features, allowing precise, realistic, and personalized simulations of disease progression. Our approach begins by leveraging large language models (LLMs) to recaption prompt for disease trajectory. Next, a controllable multi-round diffusion model simulates the disease progression state for each patient, creating realistic intermediate disease state sequence. Finally, a diffusion-based video transition generation model interpolates disease progression between these states. We validate our framework across three medical imaging domains: chest X-ray, fundus photography, and skin image. Our results demonstrate that MVG significantly outperforms baseline models in generating coherent and clinically plausible disease trajectories. Two user studies by veteran physicians, provide further validation and insights into the clinical utility of the generated sequences. MVG has the potential to assist healthcare providers in modeling disease trajectories, interpolating missing medical image data, and enhancing medical education through realistic, dynamic visualizations of disease progression.

Related papers

• Unscrambling disease progression at scale: fast inference of event permutations with optimal transport [2.9087305408570945]
  Disease progression models infer group-level temporal trajectories of change in patients' features as a chronic degenerative condition plays out. We leverage ideas from optimal transport to model disease progression as a latent permutation matrix of events belonging to the Birkhoff polytope. Experiments demonstrate the increase in speed, accuracy and robustness to noise in simulation.
  arXiv  Detail & Related papers  (2024-10-18T11:44:29Z)
• Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports [51.45762396192655]
  Multimodal large language models (MLLMs) have recently transformed many domains, significantly affecting the medical field. Notably, Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models have epitomized a paradigm shift in Artificial General Intelligence for computer vision. This study evaluated the performance of the Gemini, GPT-4, and 4 popular large models for an exhaustive evaluation across 14 medical imaging datasets.
  arXiv  Detail & Related papers  (2024-07-08T09:08:42Z)
• Probabilistic Temporal Prediction of Continuous Disease Trajectories and Treatment Effects Using Neural SDEs [6.5527554277858275]
  We present the first causal temporal framework to model the continuous temporal evolution of disease progression via Neural Differential Equations (NSDE) Our results present the first successful uncertainty-based causal Deep Learning (DL) model to accurately predict future patient MS disability (e.g. EDSS) and treatment effects.
  arXiv  Detail & Related papers  (2024-06-18T17:22:55Z)
• PIE: Simulating Disease Progression via Progressive Image Editing [27.658116659009025]
  Progressive Image Editing (PIE) enables controlled manipulation of disease-related image features. We leverage recent advancements in text-to-image generative models to simulate disease progression accurately and personalize it for each patient. PIE is the first of its kind to generate disease progression images meeting real-world standards.
  arXiv  Detail & Related papers  (2023-09-21T02:46:32Z)
• LVM-Med: Learning Large-Scale Self-Supervised Vision Models for Medical Imaging via Second-order Graph Matching [59.01894976615714]
  We introduce LVM-Med, the first family of deep networks trained on large-scale medical datasets. We have collected approximately 1.3 million medical images from 55 publicly available datasets. LVM-Med empirically outperforms a number of state-of-the-art supervised, self-supervised, and foundation models.
  arXiv  Detail & Related papers  (2023-06-20T22:21:34Z)
• Mixture of Input-Output Hidden Markov Models for Heterogeneous Disease Progression Modeling [11.768140291216769]
  We propose a hierarchical time-series model that can discover multiple disease progression dynamics. We illustrate the benefits of our model using a synthetically generated dataset and a real-world longitudinal dataset for Parkinson's disease.
  arXiv  Detail & Related papers  (2022-07-24T23:17:06Z)
• AlignTransformer: Hierarchical Alignment of Visual Regions and Disease Tags for Medical Report Generation [50.21065317817769]
  We propose an AlignTransformer framework, which includes the Align Hierarchical Attention (AHA) and the Multi-Grained Transformer (MGT) modules. Experiments on the public IU-Xray and MIMIC-CXR datasets show that the AlignTransformer can achieve results competitive with state-of-the-art methods on the two datasets.
  arXiv  Detail & Related papers  (2022-03-18T13:43:53Z)
• Context-aware Health Event Prediction via Transition Functions on Dynamic Disease Graphs [15.17817233616652]
  Many machine learning approaches assume disease representations are static in different visits of a patient. We propose a novel context-aware learning framework using transition functions on dynamic disease graphs. Experimental results on two real-world EHR datasets show that the proposed model outperforms state of the art in predicting health events.
  arXiv  Detail & Related papers  (2021-12-09T20:06:39Z)
• Many-to-One Distribution Learning and K-Nearest Neighbor Smoothing for Thoracic Disease Identification [83.6017225363714]
  deep learning has become the most powerful computer-aided diagnosis technology for improving disease identification performance. For chest X-ray imaging, annotating large-scale data requires professional domain knowledge and is time-consuming. In this paper, we propose many-to-one distribution learning (MODL) and K-nearest neighbor smoothing (KNNS) methods to improve a single model's disease identification performance.
  arXiv  Detail & Related papers  (2021-02-26T02:29:30Z)
• Learning Dynamic and Personalized Comorbidity Networks from Event Data using Deep Diffusion Processes [102.02672176520382]
  Comorbid diseases co-occur and progress via complex temporal patterns that vary among individuals. In electronic health records we can observe the different diseases a patient has, but can only infer the temporal relationship between each co-morbid condition. We develop deep diffusion processes to model "dynamic comorbidity networks"
  arXiv  Detail & Related papers  (2020-01-08T15:47:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.