MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning

• URL: http://arxiv.org/abs/2312.14574v2
• Date: Thu, 27 Jun 2024 13:29:07 GMT
• Title: MMGPL: Multimodal Medical Data Analysis with Graph Prompt Learning
• Authors: Liang Peng, Songyue Cai, Zongqian Wu, Huifang Shang, Xiaofeng Zhu, Xiaoxiao Li,
• Abstract summary: We introduce a novel prompt learning model by learning graph prompts during the fine-tuning process of multimodal large models for diagnosing neurological disorders. Specifically, we first leverage GPT-4 to obtain relevant disease concepts and compute semantic similarity between these concepts and all patches. Secondly, we reduce the weight of irrelevant patches according to the semantic similarity between each patch and disease-related concepts.
• Score: 31.21351373001379
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prompt learning has demonstrated impressive efficacy in the fine-tuning of multimodal large models to a wide range of downstream tasks. Nonetheless, applying existing prompt learning methods for the diagnosis of neurological disorder still suffers from two issues: (i) existing methods typically treat all patches equally, despite the fact that only a small number of patches in neuroimaging are relevant to the disease, and (ii) they ignore the structural information inherent in the brain connection network which is crucial for understanding and diagnosing neurological disorders. To tackle these issues, we introduce a novel prompt learning model by learning graph prompts during the fine-tuning process of multimodal large models for diagnosing neurological disorders. Specifically, we first leverage GPT-4 to obtain relevant disease concepts and compute semantic similarity between these concepts and all patches. Secondly, we reduce the weight of irrelevant patches according to the semantic similarity between each patch and disease-related concepts. Moreover, we construct a graph among tokens based on these concepts and employ a graph convolutional network layer to extract the structural information of the graph, which is used to prompt the pre-trained multimodal large models for diagnosing neurological disorders. Extensive experiments demonstrate that our method achieves superior performance for neurological disorder diagnosis compared with state-of-the-art methods and validated by clinicians.

Related papers

• Multi-modal Dynamic Graph Network: Coupling Structural and Functional Connectome for Disease Diagnosis and Classification [8.67028273829113]
  We propose a Multi-modal Dynamic Graph Convolution Network (MDGCN) for structural and functional brain network learning. Our method benefits from modeling inter-modal representations and relating attentive multi-model associations into dynamic graphs.
  arXiv  Detail & Related papers  (2022-10-25T02:41:32Z)
• NeuralSympCheck: A Symptom Checking and Disease Diagnostic Neural Model with Logic Regularization [59.15047491202254]
  symptom checking systems inquire users for their symptoms and perform a rapid and affordable medical assessment of their condition. We propose a new approach based on the supervised learning of neural models with logic regularization. Our experiments show that the proposed approach outperforms the best existing methods in the accuracy of diagnosis when the number of diagnoses and symptoms is large.
  arXiv  Detail & Related papers  (2022-06-02T07:57:17Z)
• A Prior Guided Adversarial Representation Learning and Hypergraph Perceptual Network for Predicting Abnormal Connections of Alzheimer's Disease [29.30199956567813]
  Alzheimer's disease is characterized by alterations of the brain's structural and functional connectivity. PGARL-HPN is proposed to predict abnormal brain connections using triple-modality medical images.
  arXiv  Detail & Related papers  (2021-10-12T03:10:37Z)
• Multimodal Representations Learning and Adversarial Hypergraph Fusion for Early Alzheimer's Disease Prediction [30.99183477161096]
  We propose a novel representation learning and adversarial hypergraph fusion framework for Alzheimer's disease diagnosis. Our model achieves superior performance on Alzheimer's disease detection compared with other related models.
  arXiv  Detail & Related papers  (2021-07-21T08:08:05Z)
• Applications of Epileptic Seizures Detection in Neuroimaging Modalities Using Deep Learning Techniques: Methods, Challenges, and Future Works [12.393282115173387]
  Epileptic seizures are a type of neurological disorder that affect many people worldwide. Specialist physicians and neurologists take advantage of structural and functional neuroimaging modalities to diagnose various types of epileptic seizures. One method to accelerate the accurate and fast diagnosis of epileptic seizures is to employ computer aided diagnosis systems (CADS) based on artificial intelligence (AI) and functional and structural neuroimaging modalities.
  arXiv  Detail & Related papers  (2021-05-29T12:00:39Z)
• Inheritance-guided Hierarchical Assignment for Clinical Automatic Diagnosis [50.15205065710629]
  Clinical diagnosis, which aims to assign diagnosis codes for a patient based on the clinical note, plays an essential role in clinical decision-making. We propose a novel framework to combine the inheritance-guided hierarchical assignment and co-occurrence graph propagation for clinical automatic diagnosis.
  arXiv  Detail & Related papers  (2021-01-27T13:16:51Z)
• Multimodal Gait Recognition for Neurodegenerative Diseases [38.06704951209703]
  We propose a novel hybrid model to learn the gait differences between three neurodegenerative diseases. A new correlative memory neural network architecture is designed for extracting temporal features. Compared with several state-of-the-art techniques, our proposed framework shows more accurate classification results.
  arXiv  Detail & Related papers  (2021-01-07T10:17:11Z)
• Graph-Evolving Meta-Learning for Low-Resource Medical Dialogue Generation [150.52617238140868]
  We propose low-resource medical dialogue generation to transfer the diagnostic experience from source diseases to target ones. We also develop a Graph-Evolving Meta-Learning framework that learns to evolve the commonsense graph for reasoning disease-symptom correlations in a new disease.
  arXiv  Detail & Related papers  (2020-12-22T13:20:23Z)
• A Graph Gaussian Embedding Method for Predicting Alzheimer's Disease Progression with MEG Brain Networks [59.15734147867412]
  Characterizing the subtle changes of functional brain networks associated with Alzheimer's disease (AD) is important for early diagnosis and prediction of disease progression. We developed a new deep learning method, termed multiple graph Gaussian embedding model (MG2G) We used MG2G to detect the intrinsic latent dimensionality of MEG brain networks, predict the progression of patients with mild cognitive impairment (MCI) to AD, and identify brain regions with network alterations related to MCI.
  arXiv  Detail & Related papers  (2020-05-08T02:29:24Z)
• Dynamic Graph Correlation Learning for Disease Diagnosis with Incomplete Labels [66.57101219176275]
  Disease diagnosis on chest X-ray images is a challenging multi-label classification task. We propose a Disease Diagnosis Graph Convolutional Network (DD-GCN) that presents a novel view of investigating the inter-dependency among different diseases. Our method is the first to build a graph over the feature maps with a dynamic adjacency matrix for correlation learning.
  arXiv  Detail & Related papers  (2020-02-26T17:10:48Z)
• Learning Dynamic and Personalized Comorbidity Networks from Event Data using Deep Diffusion Processes [102.02672176520382]
  Comorbid diseases co-occur and progress via complex temporal patterns that vary among individuals. In electronic health records we can observe the different diseases a patient has, but can only infer the temporal relationship between each co-morbid condition. We develop deep diffusion processes to model "dynamic comorbidity networks"
  arXiv  Detail & Related papers  (2020-01-08T15:47:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.