MedCycle: Unpaired Medical Report Generation via Cycle-Consistency

• URL: http://arxiv.org/abs/2403.13444v2
• Date: Thu, 21 Mar 2024 17:19:25 GMT
• Title: MedCycle: Unpaired Medical Report Generation via Cycle-Consistency
• Authors: Elad Hirsch, Gefen Dawidowicz, Ayellet Tal,
• Abstract summary: We introduce an innovative approach that eliminates the need for consistent labeling schemas. This approach is based on cycle-consistent mapping functions that transform image embeddings into report embeddings. It outperforms state-of-the-art results in unpaired chest X-ray report generation, demonstrating improvements in both language and clinical metrics.
• Score: 11.190146577567548
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating medical reports for X-ray images presents a significant challenge, particularly in unpaired scenarios where access to paired image-report data for training is unavailable. Previous works have typically learned a joint embedding space for images and reports, necessitating a specific labeling schema for both. We introduce an innovative approach that eliminates the need for consistent labeling schemas, thereby enhancing data accessibility and enabling the use of incompatible datasets. This approach is based on cycle-consistent mapping functions that transform image embeddings into report embeddings, coupled with report auto-encoding for medical report generation. Our model and objectives consider intricate local details and the overarching semantic context within images and reports. This approach facilitates the learning of effective mapping functions, resulting in the generation of coherent reports. It outperforms state-of-the-art results in unpaired chest X-ray report generation, demonstrating improvements in both language and clinical metrics.

Related papers

• MedRAT: Unpaired Medical Report Generation via Auxiliary Tasks [11.190146577567548]
  We propose a novel model that leverages the available information in two distinct datasets. Our model, named MedRAT, surpasses previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-07-04T13:31:47Z)
• Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generation [10.46031380503486]
  We introduce a novel method, textbfStructural textbfEntities extraction and patient indications textbfIncorporation (SEI) for chest X-ray report generation. We employ a structural entities extraction (SEE) approach to eliminate presentation-style vocabulary in reports. We propose a cross-modal fusion network to integrate information from X-ray images, similar historical cases, and patient-specific indications.
  arXiv  Detail & Related papers  (2024-05-23T01:29:47Z)
• Fine-Grained Image-Text Alignment in Medical Imaging Enables Explainable Cyclic Image-Report Generation [91.63262242041695]
  We propose a novel Adaptive patch-word Matching (AdaMatch) model to correlate chest X-ray (CXR) image regions with words in medical reports. AdaMatch exploits the fine-grained relation between adaptive patches and words to provide explanations of specific image regions with corresponding words. In order to provide explicit explainability for CXR-report generation task, we propose an AdaMatch-based bidirectional large language model for Cyclic CXR-report generation.
  arXiv  Detail & Related papers  (2023-12-13T11:47:28Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• C^2M-DoT: Cross-modal consistent multi-view medical report generation with domain transfer network [67.97926983664676]
  We propose a cross-modal consistent multi-view medical report generation with a domain transfer network (C2M-DoT) C2M-DoT substantially outperforms state-of-the-art baselines in all metrics.
  arXiv  Detail & Related papers  (2023-10-09T02:31:36Z)
• Automatic Radiology Report Generation by Learning with Increasingly Hard Negatives [23.670280341513795]
  This paper proposes a novel framework to learn discriminative image and report features. It distinguishes them from their closest peers, i.e., hard negatives. It can serve as a plug-in to readily improve existing medical report generation models.
  arXiv  Detail & Related papers  (2023-05-11T23:12:13Z)
• Dynamic Graph Enhanced Contrastive Learning for Chest X-ray Report Generation [92.73584302508907]
  We propose a knowledge graph with Dynamic structure and nodes to facilitate medical report generation with Contrastive Learning. In detail, the fundamental structure of our graph is pre-constructed from general knowledge. Each image feature is integrated with its very own updated graph before being fed into the decoder module for report generation.
  arXiv  Detail & Related papers  (2023-03-18T03:53:43Z)
• Lesion Guided Explainable Few Weak-shot Medical Report Generation [25.15493013683396]
  We propose a lesion guided explainable few weak-shot medical report generation framework. It learns correlation between seen and novel classes through visual and semantic feature alignment. It aims to generate medical reports for diseases not observed in training.
  arXiv  Detail & Related papers  (2022-11-16T07:47:29Z)
• Variational Topic Inference for Chest X-Ray Report Generation [102.04931207504173]
  Report generation for medical imaging promises to reduce workload and assist diagnosis in clinical practice. Recent work has shown that deep learning models can successfully caption natural images. We propose variational topic inference for automatic report generation.
  arXiv  Detail & Related papers  (2021-07-15T13:34:38Z)
• Learning Visual-Semantic Embeddings for Reporting Abnormal Findings on Chest X-rays [6.686095511538683]
  This work focuses on reporting abnormal findings on radiology images. We propose a method to identify abnormal findings from the reports in addition to grouping them with unsupervised clustering and minimal rules. We demonstrate that our method is able to retrieve abnormal findings and outperforms existing generation models on both clinical correctness and text generation metrics.
  arXiv  Detail & Related papers  (2020-10-06T04:18:18Z)
• Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation [107.3538598876467]
  We propose an Auxiliary Signal-Guided Knowledge-Decoder (ASGK) to mimic radiologists' working patterns. ASGK integrates internal visual feature fusion and external medical linguistic information to guide medical knowledge transfer and learning.
  arXiv  Detail & Related papers  (2020-06-06T01:00:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.