Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models

• URL: http://arxiv.org/abs/2305.03660v1
• Date: Fri, 5 May 2023 16:28:03 GMT
• Title: Retrieval Augmented Chest X-Ray Report Generation using OpenAI GPT models
• Authors: Mercy Ranjit, Gopinath Ganapathy, Ranjit Manuel, Tanuja Ganu
• Abstract summary: RAG is an approach for automated radiology report writing that leverages multimodally aligned embeddings from a contrastively pretrained vision language model. Our approach achieves better clinical metrics with a BERTScore of 0.2865 (Delta+ 25.88%) and Semb score of 0.4026 (Delta+ 6.31%)
• Score: 0.9339914898177185
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We propose Retrieval Augmented Generation (RAG) as an approach for automated radiology report writing that leverages multimodally aligned embeddings from a contrastively pretrained vision language model for retrieval of relevant candidate radiology text for an input radiology image and a general domain generative model like OpenAI text-davinci-003, gpt-3.5-turbo and gpt-4 for report generation using the relevant radiology text retrieved. This approach keeps hallucinated generations under check and provides capabilities to generate report content in the format we desire leveraging the instruction following capabilities of these generative models. Our approach achieves better clinical metrics with a BERTScore of 0.2865 ({\Delta}+ 25.88%) and Semb score of 0.4026 ({\Delta}+ 6.31%). Our approach can be broadly relevant for different clinical settings as it allows to augment the automated radiology report generation process with content relevant for that setting while also having the ability to inject user intents and requirements in the prompts as part of the report generation process to modulate the content and format of the generated reports as applicable for that clinical setting.

Related papers

• Structural Entities Extraction and Patient Indications Incorporation for Chest X-ray Report Generation [10.46031380503486]
  We introduce a novel method, textbfStructural textbfEntities extraction and patient indications textbfIncorporation (SEI) for chest X-ray report generation. We employ a structural entities extraction (SEE) approach to eliminate presentation-style vocabulary in reports. We propose a cross-modal fusion network to integrate information from X-ray images, similar historical cases, and patient-specific indications.
  arXiv  Detail & Related papers  (2024-05-23T01:29:47Z)
• Reshaping Free-Text Radiology Notes Into Structured Reports With Generative Transformers [0.29530625605275984]
  structured reporting (SR) has been recommended by various medical societies. We propose a pipeline to extract information from free-text reports. Our work aims to leverage the potential of Natural Language Processing (NLP) and Transformer-based models.
  arXiv  Detail & Related papers  (2024-03-27T18:38:39Z)
• RaDialog: A Large Vision-Language Model for Radiology Report Generation and Conversational Assistance [53.20640629352422]
  Conversational AI tools can generate and discuss clinically correct radiology reports for a given medical image. RaDialog is the first thoroughly evaluated and publicly available large vision-language model for radiology report generation and interactive dialog. Our method achieves state-of-the-art clinical correctness in report generation and shows impressive abilities in interactive tasks such as correcting reports and answering questions.
  arXiv  Detail & Related papers  (2023-11-30T16:28:40Z)
• Radiology Report Generation Using Transformers Conditioned with Non-imaging Data [55.17268696112258]
  This paper proposes a novel multi-modal transformer network that integrates chest x-ray (CXR) images and associated patient demographic information. The proposed network uses a convolutional neural network to extract visual features from CXRs and a transformer-based encoder-decoder network that combines the visual features with semantic text embeddings of patient demographic information.
  arXiv  Detail & Related papers  (2023-11-18T14:52:26Z)
• ChatRadio-Valuer: A Chat Large Language Model for Generalizable Radiology Report Generation Based on Multi-institution and Multi-system Data [115.0747462486285]
  ChatRadio-Valuer is a tailored model for automatic radiology report generation that learns generalizable representations. The clinical dataset utilized in this study encompasses a remarkable total of textbf332,673 observations. ChatRadio-Valuer consistently outperforms state-of-the-art models, especially ChatGPT (GPT-3.5-Turbo) and GPT-4 et al.
  arXiv  Detail & Related papers  (2023-10-08T17:23:17Z)
• PathLDM: Text conditioned Latent Diffusion Model for Histopathology [62.970593674481414]
  We introduce PathLDM, the first text-conditioned Latent Diffusion Model tailored for generating high-quality histopathology images. Our approach fuses image and textual data to enhance the generation process. We achieved a SoTA FID score of 7.64 for text-to-image generation on the TCGA-BRCA dataset, significantly outperforming the closest text-conditioned competitor with FID 30.1.
  arXiv  Detail & Related papers  (2023-09-01T22:08:32Z)
• An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT [80.33783969507458]
  The 'Impression' section of a radiology report is a critical basis for communication between radiologists and other physicians. Recent studies have achieved promising results in automatic impression generation using large-scale medical text data. These models often require substantial amounts of medical text data and have poor generalization performance.
  arXiv  Detail & Related papers  (2023-04-17T17:13:42Z)
• Medical Image Captioning via Generative Pretrained Transformers [57.308920993032274]
  We combine two language models, the Show-Attend-Tell and the GPT-3, to generate comprehensive and descriptive radiology records. The proposed model is tested on two medical datasets, the Open-I, MIMIC-CXR, and the general-purpose MS-COCO.
  arXiv  Detail & Related papers  (2022-09-28T10:27:10Z)
• Improving Radiology Report Generation Systems by Removing Hallucinated References to Non-existent Priors [1.1110995501996481]
  We propose two methods to remove references to priors in radiology reports. A GPT-3-based few-shot approach to rewrite medical reports without references to priors; and a BioBERT-based token classification approach to directly remove words referring to priors. We find that our re-trained model--which we call CXR-ReDonE--outperforms previous report generation methods on clinical metrics, achieving an average BERTScore of 0.2351 (2.57% absolute improvement)
  arXiv  Detail & Related papers  (2022-09-27T00:44:41Z)
• Generating Radiology Reports via Memory-driven Transformer [38.30011851429407]
  We propose to generate radiology reports with memory-driven Transformer. Experimental results on two prevailing radiology report datasets, IU X-Ray and MIMIC-CXR.
  arXiv  Detail & Related papers  (2020-10-30T04:08:03Z)
• Improving Factual Completeness and Consistency of Image-to-Text Radiology Report Generation [26.846912996765447]
  We introduce two new simple rewards to encourage the generation of factually complete and consistent radiology reports. We show that our system leads to generations that are more factually complete and consistent compared to the baselines.
  arXiv  Detail & Related papers  (2020-10-20T05:42:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.