Cross-Modal Causal Intervention for Medical Report Generation
- URL: http://arxiv.org/abs/2303.09117v4
- Date: Wed, 28 Feb 2024 08:57:09 GMT
- Title: Cross-Modal Causal Intervention for Medical Report Generation
- Authors: Weixing Chen, Yang Liu, Ce Wang, Jiarui Zhu, Shen Zhao, Guanbin Li,
Cheng-Lin Liu and Liang Lin
- Abstract summary: Medical report generation (MRG) is essential for computer-aided diagnosis and medication guidance.
Due to the spurious correlations within image-text data induced by visual and linguistic biases, it is challenging to generate accurate reports reliably describing lesion areas.
We propose a novel Visual-Linguistic Causal Intervention (VLCI) framework for MRG, which consists of a visual deconfounding module (VDM) and a linguistic deconfounding module (LDM)
- Score: 109.83549148448469
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Medical report generation (MRG) is essential for computer-aided diagnosis and
medication guidance, which can relieve the heavy burden of radiologists by
automatically generating the corresponding medical reports according to the
given radiology image. However, due to the spurious correlations within
image-text data induced by visual and linguistic biases, it is challenging to
generate accurate reports reliably describing lesion areas. Moreover, the
cross-modal confounders are usually unobservable and challenging to be
eliminated explicitly. In this paper, we aim to mitigate the cross-modal data
bias for MRG from a new perspective, i.e., cross-modal causal intervention, and
propose a novel Visual-Linguistic Causal Intervention (VLCI) framework for MRG,
which consists of a visual deconfounding module (VDM) and a linguistic
deconfounding module (LDM), to implicitly mitigate the visual-linguistic
confounders by causal front-door intervention. Specifically, due to the absence
of a generalized semantic extractor, the VDM explores and disentangles the
visual confounders from the patch-based local and global features without
expensive fine-grained annotations. Simultaneously, due to the lack of
knowledge encompassing the entire field of medicine, the LDM eliminates the
linguistic confounders caused by salient visual features and high-frequency
context without constructing a terminology database. Extensive experiments on
IU-Xray and MIMIC-CXR datasets show that our VLCI significantly outperforms the
state-of-the-art MRG methods. The code and models are available at
https://github.com/WissingChen/VLCI.
Related papers
- Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generation [10.46031380503486]
We introduce a novel method, textbfStructural textbfEntities extraction and patient indications textbfIncorporation (SEI) for chest X-ray report generation.
We employ a structural entities extraction (SEE) approach to eliminate presentation-style vocabulary in reports.
We propose a cross-modal fusion network to integrate information from X-ray images, similar historical cases, and patient-specific indications.
arXiv Detail & Related papers (2024-05-23T01:29:47Z) - Topicwise Separable Sentence Retrieval for Medical Report Generation [41.812337937025084]
We introduce a Topicwise Separable Sentence Retrieval (Teaser) for medical report generation.
To ensure comprehensive learning of both common and rare topics, we categorize queries into common and rare types, and then propose Topic Contrastive Loss.
Experiments on the MIMIC-CXR and IU X-ray datasets demonstrate that Teaser surpasses state-of-the-art models.
arXiv Detail & Related papers (2024-05-07T10:21:23Z) - SERPENT-VLM : Self-Refining Radiology Report Generation Using Vision Language Models [9.390882250428305]
Radiology Report Generation (R2Gen) demonstrates how Multi-modal Large Language Models (MLLMs) can automate the creation of accurate and coherent radiological reports.
Existing methods often hallucinate details in text-based reports that don't accurately reflect the image content.
We introduce a novel strategy, which improves the R2Gen task by integrating a self-refining mechanism into the MLLM framework.
arXiv Detail & Related papers (2024-04-27T13:46:23Z) - Medical Report Generation based on Segment-Enhanced Contrastive
Representation Learning [39.17345313432545]
We propose MSCL (Medical image with Contrastive Learning) to segment organs, abnormalities, bones, etc.
We introduce a supervised contrastive loss that assigns more weight to reports that are semantically similar to the target while training.
Experimental results demonstrate the effectiveness of our proposed model, where we achieve state-of-the-art performance on the IU X-Ray public dataset.
arXiv Detail & Related papers (2023-12-26T03:33:48Z) - Radiology Report Generation Using Transformers Conditioned with
Non-imaging Data [55.17268696112258]
This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information.
The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
arXiv Detail & Related papers (2023-11-18T14:52:26Z) - Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report
Generation [92.73584302508907]
We propose a knowledge graph with Dynamic structure and nodes to facilitate medical report generation with Contrastive Learning.
In detail, the fundamental structure of our graph is pre-constructed from general knowledge.
Each image feature is integrated with its very own updated graph before being fed into the decoder module for report generation.
arXiv Detail & Related papers (2023-03-18T03:53:43Z) - Cross-Modal Causal Relational Reasoning for Event-Level Visual Question
Answering [134.91774666260338]
Existing visual question answering methods often suffer from cross-modal spurious correlations and oversimplified event-level reasoning processes.
We propose a framework for cross-modal causal relational reasoning to address the task of event-level visual question answering.
arXiv Detail & Related papers (2022-07-26T04:25:54Z) - A Self-Guided Framework for Radiology Report Generation [10.573538773141715]
A self-guided framework (SGF) is developed to generate medical reports with annotated disease labels.
SGF uses unsupervised and supervised deep learning methods to mimic the process of human learning and writing.
Our results highlight the capacity of the proposed framework to distinguish fined-grained visual details between words.
arXiv Detail & Related papers (2022-06-19T11:09:27Z) - AlignTransformer: Hierarchical Alignment of Visual Regions and Disease
Tags for Medical Report Generation [50.21065317817769]
We propose an AlignTransformer framework, which includes the Align Hierarchical Attention (AHA) and the Multi-Grained Transformer (MGT) modules.
Experiments on the public IU-Xray and MIMIC-CXR datasets show that the AlignTransformer can achieve results competitive with state-of-the-art methods on the two datasets.
arXiv Detail & Related papers (2022-03-18T13:43:53Z) - Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report
Generation [107.3538598876467]
We propose an Auxiliary Signal-Guided Knowledge-Decoder (ASGK) to mimic radiologists' working patterns.
ASGK integrates internal visual feature fusion and external medical linguistic information to guide medical knowledge transfer and learning.
arXiv Detail & Related papers (2020-06-06T01:00:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.