X-ray Made Simple: Radiology Report Generation and Evaluation with Layman's Terms

• URL: http://arxiv.org/abs/2406.17911v3
• Date: Wed, 16 Oct 2024 21:57:58 GMT
• Title: X-ray Made Simple: Radiology Report Generation and Evaluation with Layman's Terms
• Authors: Kun Zhao, Chenghao Xiao, Chen Tang, Bohao Yang, Kai Ye, Noura Al Moubayed, Liang Zhan, Chenghua Lin,
• Abstract summary: Radiology Report Generation (RRG) has achieved significant progress with the advancements of multimodal generative models. High performance on RRG with existing lexical-based metrics (e.g. BLEU) might be more of a mirage - a model can get a high BLEU only by learning the template of reports. We un-intuitively approach this problem by proposing the Layman's RRG framework, a layman's terms-based dataset, evaluation and training framework.
• Score: 25.871814979179373
• License:
• Abstract: Radiology Report Generation (RRG) has achieved significant progress with the advancements of multimodal generative models. However, the evaluation in the domain suffers from a lack of fair and robust metrics. We reveal that, high performance on RRG with existing lexical-based metrics (e.g. BLEU) might be more of a mirage - a model can get a high BLEU only by learning the template of reports. This has become an urgent problem for RRG due to the highly patternized nature of these reports. In this work, we un-intuitively approach this problem by proposing the Layman's RRG framework, a layman's terms-based dataset, evaluation and training framework that systematically improves RRG with day-to-day language. We first contribute the translated Layman's terms dataset. Building upon the dataset, we then propose a semantics-based evaluation method, which is proved to mitigate the inflated numbers of BLEU and provides fairer evaluation. Last, we show that training on the layman's terms dataset encourages models to focus on the semantics of the reports, as opposed to overfitting to learning the report templates. We reveal a promising scaling law between the number of training examples and semantics gain provided by our dataset, compared to the inverse pattern brought by the original formats. Our code is available at \url{https://github.com/hegehongcha/LaymanRRG}.

Related papers

• CXPMRG-Bench: Pre-training and Benchmarking for X-ray Medical Report Generation on CheXpert Plus Dataset [14.911363203907008]
  X-ray image-based medical report generation can significantly reduce diagnostic burdens and patient wait times. We conduct a comprehensive benchmarking of existing mainstream X-ray report generation models and large language models (LLMs) on the CheXpert Plus dataset. We propose a large model for the X-ray image report generation using a multi-stage pre-training strategy, including self-supervised autoregressive generation and Xray-report contrastive learning.
  arXiv  Detail & Related papers  (2024-10-01T04:07:01Z)
• GLIMMER: Incorporating Graph and Lexical Features in Unsupervised Multi-Document Summarization [13.61818620609812]
  We propose a lightweight yet effective unsupervised approach called GLIMMER: a Graph and LexIcal features based unsupervised Multi-docuMEnt summaRization approach. It first constructs a sentence graph from the source documents, then automatically identifies semantic clusters by mining low-level features from raw texts. Experiments conducted on Multi-News, Multi-XScience and DUC-2004 demonstrate that our approach outperforms existing unsupervised approaches.
  arXiv  Detail & Related papers  (2024-08-19T16:01:48Z)
• Relation Extraction with Fine-Tuned Large Language Models in Retrieval Augmented Generation Frameworks [0.0]
  Relation Extraction (RE) is crucial for converting unstructured data into structured formats like Knowledge Graphs (KGs) Recent studies leveraging pre-trained language models (PLMs) have shown significant success in this area. This work explores the performance of fine-tuned LLMs and their integration into the Retrieval Augmented-based (RAG) RE approach.
  arXiv  Detail & Related papers  (2024-06-20T21:27:57Z)
• MRScore: Evaluating Radiology Report Generation with LLM-based Reward System [39.54237580336297]
  This paper introduces MRScore, an automatic evaluation metric tailored for radiology report generation by leveraging Large Language Models (LLMs) To address this challenge, we collaborated with radiologists to develop a framework that guides LLMs for radiology report evaluation, ensuring alignment with human analysis. Our experiments demonstrate MRScore's higher correlation with human judgments and superior performance in model selection compared to traditional metrics.
  arXiv  Detail & Related papers  (2024-04-27T04:42:45Z)
• ExaRanker-Open: Synthetic Explanation for IR using Open-Source LLMs [60.81649785463651]
  We introduce ExaRanker-Open, where we adapt and explore the use of open-source language models to generate explanations. Our findings reveal that incorporating explanations consistently enhances neural rankers, with benefits escalating as the LLM size increases.
  arXiv  Detail & Related papers  (2024-02-09T11:23:14Z)
• See, Say, and Segment: Teaching LMMs to Overcome False Premises [67.36381001664635]
  We propose a cascading and joint training approach for LMMs to solve this task. Our resulting model can "see" by detecting whether objects are present in an image, "say" by telling the user if they are not, and finally "segment" by outputting the mask of the desired objects if they exist.
  arXiv  Detail & Related papers  (2023-12-13T18:58:04Z)
• DORE: Document Ordered Relation Extraction based on Generative Framework [56.537386636819626]
  This paper investigates the root cause of the underwhelming performance of the existing generative DocRE models. We propose to generate a symbolic and ordered sequence from the relation matrix which is deterministic and easier for model to learn. Experimental results on four datasets show that our proposed method can improve the performance of the generative DocRE models.
  arXiv  Detail & Related papers  (2022-10-28T11:18:10Z)
• A Closer Look at Debiased Temporal Sentence Grounding in Videos: Dataset, Metric, and Approach [53.727460222955266]
  Temporal Sentence Grounding in Videos (TSGV) aims to ground a natural language sentence in an untrimmed video. Recent studies have found that current benchmark datasets may have obvious moment annotation biases. We introduce a new evaluation metric "dR@n,IoU@m" that discounts the basic recall scores to alleviate the inflating evaluation caused by biased datasets.
  arXiv  Detail & Related papers  (2022-03-10T08:58:18Z)
• Automatically Generating Counterfactuals for Relation Exaction [18.740447044960796]
  relation extraction (RE) is a fundamental task in natural language processing. Current deep neural models have achieved high accuracy but are easily affected by spurious correlations. We develop a novel approach to derive contextual counterfactuals for entities.
  arXiv  Detail & Related papers  (2022-02-22T04:46:10Z)
• Coreference Reasoning in Machine Reading Comprehension [100.75624364257429]
  We show that coreference reasoning in machine reading comprehension is a greater challenge than was earlier thought. We propose a methodology for creating reading comprehension datasets that better reflect the challenges of coreference reasoning. This allows us to show an improvement in the reasoning abilities of state-of-the-art models across various MRC datasets.
  arXiv  Detail & Related papers  (2020-12-31T12:18:41Z)
• Learning from Context or Names? An Empirical Study on Neural Relation Extraction [112.06614505580501]
  We study the effect of two main information sources in text: textual context and entity mentions (names) We propose an entity-masked contrastive pre-training framework for relation extraction (RE) Our framework can improve the effectiveness and robustness of neural models in different RE scenarios.
  arXiv  Detail & Related papers  (2020-10-05T11:21:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.