REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records
Analysis via Large Language Models
- URL: http://arxiv.org/abs/2402.07016v1
- Date: Sat, 10 Feb 2024 18:27:28 GMT
- Title: REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records
Analysis via Large Language Models
- Authors: Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang
Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, Chengwei Pan
- Abstract summary: Existing models often lack the medical context relevent to clinical tasks, prompting the incorporation of external knowledge.
We propose REALM, a Retrieval-Augmented Generation (RAG) driven framework to enhance multimodal EHR representations.
Our experiments on MIMIC-III mortality and readmission tasks showcase the superior performance of our REALM framework over baselines.
- Score: 19.62552013839689
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The integration of multimodal Electronic Health Records (EHR) data has
significantly improved clinical predictive capabilities. Leveraging clinical
notes and multivariate time-series EHR, existing models often lack the medical
context relevent to clinical tasks, prompting the incorporation of external
knowledge, particularly from the knowledge graph (KG). Previous approaches with
KG knowledge have primarily focused on structured knowledge extraction,
neglecting unstructured data modalities and semantic high dimensional medical
knowledge. In response, we propose REALM, a Retrieval-Augmented Generation
(RAG) driven framework to enhance multimodal EHR representations that address
these limitations. Firstly, we apply Large Language Model (LLM) to encode long
context clinical notes and GRU model to encode time-series EHR data. Secondly,
we prompt LLM to extract task-relevant medical entities and match entities in
professionally labeled external knowledge graph (PrimeKG) with corresponding
medical knowledge. By matching and aligning with clinical standards, our
framework eliminates hallucinations and ensures consistency. Lastly, we propose
an adaptive multimodal fusion network to integrate extracted knowledge with
multimodal EHR data. Our extensive experiments on MIMIC-III mortality and
readmission tasks showcase the superior performance of our REALM framework over
baselines, emphasizing the effectiveness of each module. REALM framework
contributes to refining the use of multimodal EHR data in healthcare and
bridging the gap with nuanced medical context essential for informed clinical
predictions.
Related papers
- MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models [11.798375238713488]
MEDFuse is a framework that integrates structured and unstructured medical data.
It achieves over 90% F1 score in the 10-disease multi-label classification task.
arXiv Detail & Related papers (2024-07-17T04:17:09Z) - medIKAL: Integrating Knowledge Graphs as Assistants of LLMs for Enhanced Clinical Diagnosis on EMRs [13.806201934732321]
medIKAL combines Large Language Models (LLMs) with knowledge graphs (KGs) to enhance diagnostic capabilities.
medIKAL assigns weighted importance to entities in medical records based on their type, enabling precise localization of candidate diseases within KGs.
We validated medIKAL's effectiveness through extensive experiments on a newly introduced open-sourced Chinese EMR dataset.
arXiv Detail & Related papers (2024-06-20T13:56:52Z) - GAMedX: Generative AI-based Medical Entity Data Extractor Using Large Language Models [1.123722364748134]
This paper introduces GAMedX, a Named Entity Recognition (NER) approach utilizing Large Language Models (LLMs)
The methodology integrates open-source LLMs for NER, utilizing chained prompts and Pydantic schemas for structured output to navigate the complexities of specialized medical jargon.
The findings reveal significant ROUGE F1 score on one of the evaluation datasets with an accuracy of 98%.
arXiv Detail & Related papers (2024-05-31T02:53:22Z) - EMERGE: Integrating RAG for Improved Multimodal EHR Predictive Modeling [22.94521527609479]
EMERGE is a Retrieval-Augmented Generation driven framework aimed at enhancing multimodal EHR predictive modeling.
Our approach extracts entities from both time-series data and clinical notes by prompting Large Language Models.
The extracted knowledge is then used to generate task-relevant summaries of patients' health statuses.
arXiv Detail & Related papers (2024-05-27T10:53:15Z) - AI Hospital: Benchmarking Large Language Models in a Multi-agent Medical Interaction Simulator [69.51568871044454]
We introduce textbfAI Hospital, a framework simulating dynamic medical interactions between emphDoctor as player and NPCs.
This setup allows for realistic assessments of LLMs in clinical scenarios.
We develop the Multi-View Medical Evaluation benchmark, utilizing high-quality Chinese medical records and NPCs.
arXiv Detail & Related papers (2024-02-15T06:46:48Z) - Next Visit Diagnosis Prediction via Medical Code-Centric Multimodal Contrastive EHR Modelling with Hierarchical Regularisation [0.0]
We propose NECHO, a novel medical code-centric multimodal contrastive EHR learning framework with hierarchical regularisation.
First, we integrate multifaceted information encompassing medical codes, demographics, and clinical notes using a tailored network design.
We also regularise modality-specific encoders using a parental level information in medical ontology to learn hierarchical structure of EHR data.
arXiv Detail & Related papers (2024-01-22T01:58:32Z) - XAI for In-hospital Mortality Prediction via Multimodal ICU Data [57.73357047856416]
We propose an efficient, explainable AI solution for predicting in-hospital mortality via multimodal ICU data.
We employ multimodal learning in our framework, which can receive heterogeneous inputs from clinical data and make decisions.
Our framework can be easily transferred to other clinical tasks, which facilitates the discovery of crucial factors in healthcare research.
arXiv Detail & Related papers (2023-12-29T14:28:04Z) - Learnable Weight Initialization for Volumetric Medical Image Segmentation [66.3030435676252]
We propose a learnable weight-based hybrid medical image segmentation approach.
Our approach is easy to integrate into any hybrid model and requires no external training data.
Experiments on multi-organ and lung cancer segmentation tasks demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2023-06-15T17:55:05Z) - Towards Medical Artificial General Intelligence via Knowledge-Enhanced
Multimodal Pretraining [121.89793208683625]
Medical artificial general intelligence (MAGI) enables one foundation model to solve different medical tasks.
We propose a new paradigm called Medical-knedge-enhanced mulTimOdal pretRaining (MOTOR)
arXiv Detail & Related papers (2023-04-26T01:26:19Z) - Competence-based Multimodal Curriculum Learning for Medical Report
Generation [98.10763792453925]
We propose a Competence-based Multimodal Curriculum Learning framework ( CMCL) to alleviate the data bias and make best use of available data.
Specifically, CMCL simulates the learning process of radiologists and optimize the model in a step by step manner.
Experiments on the public IU-Xray and MIMIC-CXR datasets show that CMCL can be incorporated into existing models to improve their performance.
arXiv Detail & Related papers (2022-06-24T08:16:01Z) - Cross-Modal Information Maximization for Medical Imaging: CMIM [62.28852442561818]
In hospitals, data are siloed to specific information systems that make the same information available under different modalities.
This offers unique opportunities to obtain and use at train-time those multiple views of the same information that might not always be available at test-time.
We propose an innovative framework that makes the most of available data by learning good representations of a multi-modal input that are resilient to modality dropping at test-time.
arXiv Detail & Related papers (2020-10-20T20:05:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.