Lesion Guided Explainable Few Weak-shot Medical Report Generation
- URL: http://arxiv.org/abs/2211.08732v2
- Date: Thu, 17 Nov 2022 06:48:04 GMT
- Title: Lesion Guided Explainable Few Weak-shot Medical Report Generation
- Authors: Jinghan Sun, Dong Wei, Liansheng Wang, and Yefeng Zheng
- Abstract summary: We propose a lesion guided explainable few weak-shot medical report generation framework.
It learns correlation between seen and novel classes through visual and semantic feature alignment.
It aims to generate medical reports for diseases not observed in training.
- Score: 25.15493013683396
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Medical images are widely used in clinical practice for diagnosis.
Automatically generating interpretable medical reports can reduce radiologists'
burden and facilitate timely care. However, most existing approaches to
automatic report generation require sufficient labeled data for training. In
addition, the learned model can only generate reports for the training classes,
lacking the ability to adapt to previously unseen novel diseases. To this end,
we propose a lesion guided explainable few weak-shot medical report generation
framework that learns correlation between seen and novel classes through visual
and semantic feature alignment, aiming to generate medical reports for diseases
not observed in training. It integrates a lesion-centric feature extractor and
a Transformer-based report generation module. Concretely, the lesion-centric
feature extractor detects the abnormal regions and learns correlations between
seen and novel classes with multi-view (visual and lexical) embeddings. Then,
features of the detected regions and corresponding embeddings are concatenated
as multi-view input to the report generation module for explainable report
generation, including text descriptions and corresponding abnormal regions
detected in the images. We conduct experiments on FFA-IR, a dataset providing
explainable annotations, showing that our framework outperforms others on
report generation for novel diseases.
Related papers
- Contrastive Learning with Counterfactual Explanations for Radiology Report Generation [83.30609465252441]
We propose a textbfCountertextbfFactual textbfExplanations-based framework (CoFE) for radiology report generation.
Counterfactual explanations serve as a potent tool for understanding how decisions made by algorithms can be changed by asking what if'' scenarios.
Experiments on two benchmarks demonstrate that leveraging the counterfactual explanations enables CoFE to generate semantically coherent and factually complete reports.
arXiv Detail & Related papers (2024-07-19T17:24:25Z) - MedCycle: Unpaired Medical Report Generation via Cycle-Consistency [11.190146577567548]
We introduce an innovative approach that eliminates the need for consistent labeling schemas.
This approach is based on cycle-consistent mapping functions that transform image embeddings into report embeddings.
It outperforms state-of-the-art results in unpaired chest X-ray report generation, demonstrating improvements in both language and clinical metrics.
arXiv Detail & Related papers (2024-03-20T09:40:11Z) - Controllable Chest X-Ray Report Generation from Longitudinal
Representations [13.151444796296868]
One strategy to speed up reporting is to integrate automated reporting systems.
Previous approaches to automated radiology reporting generally do not provide the prior study as input.
We introduce two novel aspects: (1) longitudinal learning -- we propose a method to align, leverage the current and prior scan information into a joint longitudinal representation which can be provided to the multimodal report generation model; (2) sentence-anatomy dropout -- a training strategy for controllability in which the report generator model is trained to predict only sentences from the original report which correspond to the subset of anatomical regions given as input.
arXiv Detail & Related papers (2023-10-09T17:22:58Z) - Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report
Generation [92.73584302508907]
We propose a knowledge graph with Dynamic structure and nodes to facilitate medical report generation with Contrastive Learning.
In detail, the fundamental structure of our graph is pre-constructed from general knowledge.
Each image feature is integrated with its very own updated graph before being fed into the decoder module for report generation.
arXiv Detail & Related papers (2023-03-18T03:53:43Z) - Cross-Modal Causal Intervention for Medical Report Generation [109.83549148448469]
Medical report generation (MRG) is essential for computer-aided diagnosis and medication guidance.
Due to the spurious correlations within image-text data induced by visual and linguistic biases, it is challenging to generate accurate reports reliably describing lesion areas.
We propose a novel Visual-Linguistic Causal Intervention (VLCI) framework for MRG, which consists of a visual deconfounding module (VDM) and a linguistic deconfounding module (LDM)
arXiv Detail & Related papers (2023-03-16T07:23:55Z) - Weakly Supervised Contrastive Learning for Chest X-Ray Report Generation [3.3978173451092437]
Radiology report generation aims at generating descriptive text from radiology images automatically.
A typical setting consists of training encoder-decoder models on image-report pairs with a cross entropy loss.
We propose a novel weakly supervised contrastive loss for medical report generation.
arXiv Detail & Related papers (2021-09-25T00:06:23Z) - Variational Topic Inference for Chest X-Ray Report Generation [102.04931207504173]
Report generation for medical imaging promises to reduce workload and assist diagnosis in clinical practice.
Recent work has shown that deep learning models can successfully caption natural images.
We propose variational topic inference for automatic report generation.
arXiv Detail & Related papers (2021-07-15T13:34:38Z) - Unifying Relational Sentence Generation and Retrieval for Medical Image
Report Composition [142.42920413017163]
Current methods often generate the most common sentences due to dataset bias for individual case.
We propose a novel framework that unifies template retrieval and sentence generation to handle both common and rare abnormality.
arXiv Detail & Related papers (2021-01-09T04:33:27Z) - Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on
Chest X-rays [6.686095511538683]
This work focuses on reporting abnormal findings on radiology images.
We propose a method to identify abnormal findings from the reports in addition to grouping them with unsupervised clustering and minimal rules.
We demonstrate that our method is able to retrieve abnormal findings and outperforms existing generation models on both clinical correctness and text generation metrics.
arXiv Detail & Related papers (2020-10-06T04:18:18Z) - Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report
Generation [107.3538598876467]
We propose an Auxiliary Signal-Guided Knowledge-Decoder (ASGK) to mimic radiologists' working patterns.
ASGK integrates internal visual feature fusion and external medical linguistic information to guide medical knowledge transfer and learning.
arXiv Detail & Related papers (2020-06-06T01:00:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.