MEIT: Multi-Modal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation
- URL: http://arxiv.org/abs/2403.04945v3
- Date: Tue, 18 Jun 2024 07:15:09 GMT
- Title: MEIT: Multi-Modal Electrocardiogram Instruction Tuning on Large Language Models for Report Generation
- Authors: Zhongwei Wan, Che Liu, Xin Wang, Chaofan Tao, Hui Shen, Zhenwu Peng, Jie Fu, Rossella Arcucci, Huaxiu Yao, Mi Zhang,
- Abstract summary: Electrocardiogram (ECG) is the primary non-invasive diagnostic tool for monitoring cardiac conditions.
Recent studies have concentrated on classifying cardiac conditions using ECG data but have overlooked ECG report generation.
We propose the Multimodal ECG Instruction Tuning (MEIT) framework, the first attempt to tackle ECG report generation with LLMs and multimodal instructions.
- Score: 41.324530807795256
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Electrocardiogram (ECG) is the primary non-invasive diagnostic tool for monitoring cardiac conditions and is crucial in assisting clinicians. Recent studies have concentrated on classifying cardiac conditions using ECG data but have overlooked ECG report generation, which is time-consuming and requires clinical expertise. To automate ECG report generation and ensure its versatility, we propose the Multimodal ECG Instruction Tuning (MEIT) framework, the first attempt to tackle ECG report generation with LLMs and multimodal instructions. To facilitate future research, we establish a benchmark to evaluate MEIT with various LLMs backbones across two large-scale ECG datasets. Our approach uniquely aligns the representations of the ECG signal and the report, and we conduct extensive experiments to benchmark MEIT with nine open-source LLMs using more than 800,000 ECG reports. MEIT's results underscore the superior performance of instruction-tuned LLMs, showcasing their proficiency in quality report generation, zero-shot capabilities, and resilience to signal perturbation. These findings emphasize the efficacy of our MEIT framework and its potential for real-world clinical application.
Related papers
- ECG Semantic Integrator (ESI): A Foundation ECG Model Pretrained with LLM-Enhanced Cardiological Text [14.06147507373525]
This study introduces a new multimodal contrastive pretaining framework that aims to improve the quality and robustness of learned representations of 12-lead ECG signals.
Our framework comprises two key components, including Cardio Query Assistant (CQA) and ECG Semantics Integrator(ESI)
arXiv Detail & Related papers (2024-05-26T06:45:39Z) - Zero-Shot ECG Classification with Multimodal Learning and Test-time Clinical Knowledge Enhancement [10.611952462532908]
Multimodal ECG Representation Learning (MERL) is capable of performing zero-shot ECG classification with text prompts.
We propose the Clinical Knowledge Enhanced Prompt Engineering (CKEPE) approach to exploit external expert-verified clinical knowledge databases.
MERL achieves an average AUC score of 75.2% in zero-shot classification (without training data), 3.2% higher than linear probed eSSL methods with 10% annotated training data, averaged across all six datasets.
arXiv Detail & Related papers (2024-03-11T12:28:55Z) - ECG-SL: Electrocardiogram(ECG) Segment Learning, a deep learning method
for ECG signal [19.885905393439014]
We propose a novel ECG-Segment based Learning (ECG-SL) framework to explicitly model the periodic nature of ECG signals.
Based on the structural features, a temporal model is designed to learn the temporal information for various clinical tasks.
The proposed method outperforms the baseline model and shows competitive performances compared with task-specific methods in three clinical applications.
arXiv Detail & Related papers (2023-10-01T23:17:55Z) - Multi-scale Cross-restoration Framework for Electrocardiogram Anomaly
Detection [33.48389041651675]
Electrocardiogram (ECG) is a widely used diagnostic tool for detecting heart conditions.
Rare cardiac diseases may be underdiagnosed using traditional ECG analysis, considering that no training dataset can exhaust all possible cardiac disorders.
This paper proposes using anomaly detection to identify any unhealthy status, with normal ECGs solely for training.
arXiv Detail & Related papers (2023-08-03T09:16:57Z) - PulseNet: Deep Learning ECG-signal classification using random
augmentation policy and continous wavelet transform for canines [46.09869227806991]
evaluating canine electrocardiograms (ECG) require skilled veterinarians.
Current availability of veterinary cardiologists for ECG interpretation and diagnostic support is limited.
We implement a deep convolutional neural network (CNN) approach for classifying canine electrocardiogram sequences as either normal or abnormal.
arXiv Detail & Related papers (2023-05-17T09:06:39Z) - Frozen Language Model Helps ECG Zero-Shot Learning [12.974685769614062]
We propose Multimodal ECG-Text Self-supervised pre-training (METS)
We use a trainable ECG encoder and a frozen language model to embed paired ECG and automatically machine-generated clinical reports separately.
In downstream classification tasks, METS achieves around 10% improvement in performance without using any annotated data.
arXiv Detail & Related papers (2023-03-22T05:01:14Z) - SEVGGNet-LSTM: a fused deep learning model for ECG classification [38.747030782394646]
The input ECG signals are firstly segmented and normalized, and then fed into the combined VGG and LSTM network for feature extraction and classification.
An attention mechanism (SE block) is embedded into the core network for increasing the weight of important features.
arXiv Detail & Related papers (2022-10-31T07:36:48Z) - Leveraging Statistical Shape Priors in GAN-based ECG Synthesis [3.3482093430607267]
We propose a novel approach for ECG signal generation using Generative Adversarial Networks (GANs) and statistical ECG data modeling.
Our approach leverages prior knowledge about ECG dynamics to synthesize realistic signals, addressing the complex dynamics of ECG signals.
Our results demonstrate that our approach, which models temporal and amplitude variations of ECG signals as 2-D shapes, generates more realistic signals compared to state-of-the-art GAN based generation baselines.
arXiv Detail & Related papers (2022-10-22T18:06:11Z) - Cross-modal Clinical Graph Transformer for Ophthalmic Report Generation [116.87918100031153]
We propose a Cross-modal clinical Graph Transformer (CGT) for ophthalmic report generation (ORG)
CGT injects clinical relation triples into the visual features as prior knowledge to drive the decoding procedure.
Experiments on the large-scale FFA-IR benchmark demonstrate that the proposed CGT is able to outperform previous benchmark methods.
arXiv Detail & Related papers (2022-06-04T13:16:30Z) - ECG-DelNet: Delineation of Ambulatory Electrocardiograms with Mixed
Quality Labeling Using Neural Networks [69.25956542388653]
Deep learning (DL) algorithms are gaining weight in academic and industrial settings.
We demonstrate DL can be successfully applied to low interpretative tasks by embedding ECG detection and delineation onto a segmentation framework.
The model was trained using PhysioNet's QT database, comprised of 105 ambulatory ECG recordings.
arXiv Detail & Related papers (2020-05-11T16:29:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.