Rethinking Medical Report Generation: Disease Revealing Enhancement with
Knowledge Graph
- URL: http://arxiv.org/abs/2307.12526v1
- Date: Mon, 24 Jul 2023 04:56:23 GMT
- Title: Rethinking Medical Report Generation: Disease Revealing Enhancement with
Knowledge Graph
- Authors: Yixin Wang, Zihao Lin, Haoyu Dong
- Abstract summary: Knowledge Graph (KG) plays a crucial role in Medical Report Generation (MRG) because it reveals the relations among diseases.
In this study, we establish a complete KG on chest X-ray imaging that includes 137 types of diseases and abnormalities.
To mitigate this problem, we introduce a novel augmentation strategy that enhances the representation of disease types in the tail-end of the distribution.
- Score: 20.48510969557314
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Knowledge Graph (KG) plays a crucial role in Medical Report Generation (MRG)
because it reveals the relations among diseases and thus can be utilized to
guide the generation process. However, constructing a comprehensive KG is
labor-intensive and its applications on the MRG process are under-explored. In
this study, we establish a complete KG on chest X-ray imaging that includes 137
types of diseases and abnormalities. Based on this KG, we find that the current
MRG data sets exhibit a long-tailed problem in disease distribution. To
mitigate this problem, we introduce a novel augmentation strategy that enhances
the representation of disease types in the tail-end of the distribution. We
further design a two-stage MRG approach, where a classifier is first trained to
detect whether the input images exhibit any abnormalities. The classified
images are then independently fed into two transformer-based generators,
namely, ``disease-specific generator" and ``disease-free generator" to generate
the corresponding reports. To enhance the clinical evaluation of whether the
generated reports correctly describe the diseases appearing in the input image,
we propose diverse sensitivity (DS), a new metric that checks whether generated
diseases match ground truth and measures the diversity of all generated
diseases. Results show that the proposed two-stage generation framework and
augmentation strategies improve DS by a considerable margin, indicating a
notable reduction in the long-tailed problem associated with under-represented
diseases.
Related papers
- AutoRG-Brain: Grounded Report Generation for Brain MRI [57.22149878985624]
Radiologists are tasked with interpreting a large number of images in a daily base, with the responsibility of generating corresponding reports.
This demanding workload elevates the risk of human error, potentially leading to treatment delays, increased healthcare costs, revenue loss, and operational inefficiencies.
We initiate a series of work on grounded Automatic Report Generation (AutoRG)
This system supports the delineation of brain structures, the localization of anomalies, and the generation of well-organized findings.
arXiv Detail & Related papers (2024-07-23T17:50:00Z) - Potential of Multimodal Large Language Models for Data Mining of Medical Images and Free-text Reports [51.45762396192655]
Multimodal large language models (MLLMs) have recently transformed many domains, significantly affecting the medical field. Notably, Gemini-Vision-series (Gemini) and GPT-4-series (GPT-4) models have epitomized a paradigm shift in Artificial General Intelligence for computer vision.
This study evaluated the performance of the Gemini, GPT-4, and 4 popular large models for an exhaustive evaluation across 14 medical imaging datasets.
arXiv Detail & Related papers (2024-07-08T09:08:42Z) - Dia-LLaMA: Towards Large Language Model-driven CT Report Generation [4.634780391920529]
We propose Dia-LLaMA, a framework to adapt the LLaMA2-7B for CT report generation by incorporating diagnostic information as guidance prompts.
Considering the high dimension of CT, we leverage a pre-trained ViT3D with perceiver to extract the visual information.
To tailor the LLM for report generation and emphasize abnormality, we extract additional diagnostic information by referring to a disease prototype memory bank.
arXiv Detail & Related papers (2024-03-25T03:02:51Z) - Complex Organ Mask Guided Radiology Report Generation [13.96983438709763]
We propose the Complex Organ Mask Guided (termed as COMG) report generation model.
We leverage prior knowledge of the disease corresponding to each organ in the fusion process to enhance the disease identification phase.
Results on two public datasets show that COMG achieves a 11.4% and 9.7% improvement in terms of BLEU@4 scores over the SOTA model KiUT.
arXiv Detail & Related papers (2023-11-04T05:34:24Z) - PromptMRG: Diagnosis-Driven Prompts for Medical Report Generation [7.508437260320598]
We propose diagnosis-driven prompts for medical report generation (PromptMRG)
PromptMRG is based on encoder-decoder architecture with an extra disease classification branch.
Cross-modal feature enhancement retrieves similar reports from the database to assist the diagnosis of a query image.
arXiv Detail & Related papers (2023-08-24T07:10:31Z) - Cross-Modal Causal Intervention for Medical Report Generation [109.83549148448469]
Medical report generation (MRG) is essential for computer-aided diagnosis and medication guidance.
Due to the spurious correlations within image-text data induced by visual and linguistic biases, it is challenging to generate accurate reports reliably describing lesion areas.
We propose a novel Visual-Linguistic Causal Intervention (VLCI) framework for MRG, which consists of a visual deconfounding module (VDM) and a linguistic deconfounding module (LDM)
arXiv Detail & Related papers (2023-03-16T07:23:55Z) - A Self-Guided Framework for Radiology Report Generation [10.573538773141715]
A self-guided framework (SGF) is developed to generate medical reports with annotated disease labels.
SGF uses unsupervised and supervised deep learning methods to mimic the process of human learning and writing.
Our results highlight the capacity of the proposed framework to distinguish fined-grained visual details between words.
arXiv Detail & Related papers (2022-06-19T11:09:27Z) - AlignTransformer: Hierarchical Alignment of Visual Regions and Disease
Tags for Medical Report Generation [50.21065317817769]
We propose an AlignTransformer framework, which includes the Align Hierarchical Attention (AHA) and the Multi-Grained Transformer (MGT) modules.
Experiments on the public IU-Xray and MIMIC-CXR datasets show that the AlignTransformer can achieve results competitive with state-of-the-art methods on the two datasets.
arXiv Detail & Related papers (2022-03-18T13:43:53Z) - Factored Attention and Embedding for Unstructured-view Topic-related
Ultrasound Report Generation [70.7778938191405]
We propose a novel factored attention and embedding model (termed FAE-Gen) for the unstructured-view topic-related ultrasound report generation.
The proposed FAE-Gen mainly consists of two modules, i.e., view-guided factored attention and topic-oriented factored embedding, which capture the homogeneous and heterogeneous morphological characteristic across different views.
arXiv Detail & Related papers (2022-03-12T15:24:03Z) - Dynamic Graph Correlation Learning for Disease Diagnosis with Incomplete
Labels [66.57101219176275]
Disease diagnosis on chest X-ray images is a challenging multi-label classification task.
We propose a Disease Diagnosis Graph Convolutional Network (DD-GCN) that presents a novel view of investigating the inter-dependency among different diseases.
Our method is the first to build a graph over the feature maps with a dynamic adjacency matrix for correlation learning.
arXiv Detail & Related papers (2020-02-26T17:10:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.