FactReranker: Fact-guided Reranker for Faithful Radiology Report Summarization

• URL: http://arxiv.org/abs/2303.08335v2
• Date: Thu, 16 Mar 2023 03:09:09 GMT
• Title: FactReranker: Fact-guided Reranker for Faithful Radiology Report Summarization
• Authors: Qianqian Xie, Jiayu Zhou, Yifan Peng, Fei Wang
• Abstract summary: We propose FactReranker, which learns to choose the best summary from all candidates based on their estimated factual consistency score. We decompose the fact-guided reranker into the factual knowledge graph generation and the factual scorer. Experimental results on two benchmark datasets demonstrate the superiority of our method in generating summaries with higher factual consistency scores.
• Score: 42.7555185736215
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Automatic radiology report summarization is a crucial clinical task, whose key challenge is to maintain factual accuracy between produced summaries and ground truth radiology findings. Existing research adopts reinforcement learning to directly optimize factual consistency metrics such as CheXBert or RadGraph score. However, their decoding method using greedy search or beam search considers no factual consistency when picking the optimal candidate, leading to limited factual consistency improvement. To address it, we propose a novel second-stage summarizing approach FactReranker, the first attempt that learns to choose the best summary from all candidates based on their estimated factual consistency score. We propose to extract medical facts of the input medical report, its gold summary, and candidate summaries based on the RadGraph schema and design the fact-guided reranker to efficiently incorporate the extracted medical facts for selecting the optimal summary. We decompose the fact-guided reranker into the factual knowledge graph generation and the factual scorer, which allows the reranker to model the mapping between the medical facts of the input text and its gold summary, thus can select the optimal summary even the gold summary can't be observed during inference. We also present a fact-based ranking metric (RadMRR) for measuring the ability of the reranker on selecting factual consistent candidates. Experimental results on two benchmark datasets demonstrate the superiority of our method in generating summaries with higher factual consistency scores when compared with existing methods.

Related papers

• Improving Factual Error Correction for Abstractive Summarization via Data Distillation and Conditional-generation Cloze [11.589564922148913]
  We first propose a novel factual error correction model FactCloze based on a conditional-generation cloze task. We then propose a data distillation method to generate a more faithful summarization dataset SummDSC.
  arXiv  Detail & Related papers  (2024-02-13T16:35:48Z)
• Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling [56.70682379371534]
  We show that our approach vastly outperforms prior methods in correcting erroneous summaries. Our model -- FactEdit -- improves factuality scores by over 11 points on CNN/DM and over 31 points on XSum.
  arXiv  Detail & Related papers  (2022-10-22T07:16:19Z)
• Word Graph Guided Summarization for Radiology Findings [24.790502861602075]
  We propose a novel method for automatic impression generation, where a word graph is constructed from the findings to record the critical words and their relations. A Word Graph guided Summarization model (WGSum) is designed to generate impressions with the help of the word graph. Experimental results on two datasets, OpenI and MIMIC-CXR, confirm the validity and effectiveness of our proposed approach.
  arXiv  Detail & Related papers  (2021-12-18T13:20:18Z)
• Weakly Supervised Contrastive Learning for Chest X-Ray Report Generation [3.3978173451092437]
  Radiology report generation aims at generating descriptive text from radiology images automatically. A typical setting consists of training encoder-decoder models on image-report pairs with a cross entropy loss. We propose a novel weakly supervised contrastive loss for medical report generation.
  arXiv  Detail & Related papers  (2021-09-25T00:06:23Z)
• Investigating Crowdsourcing Protocols for Evaluating the Factual Consistency of Summaries [59.27273928454995]
  Current pre-trained models applied to summarization are prone to factual inconsistencies which misrepresent the source text or introduce extraneous information. We create a crowdsourcing evaluation framework for factual consistency using the rating-based Likert scale and ranking-based Best-Worst Scaling protocols. We find that ranking-based protocols offer a more reliable measure of summary quality across datasets, while the reliability of Likert ratings depends on the target dataset and the evaluation design.
  arXiv  Detail & Related papers  (2021-09-19T19:05:00Z)
• Improving Factual Consistency of Abstractive Summarization via Question Answering [25.725873545789046]
  We present an approach to address factual consistency in summarization. We first propose an efficient automatic evaluation metric to measure factual consistency. We then propose a novel learning algorithm that maximizes the proposed metric during model training.
  arXiv  Detail & Related papers  (2021-05-10T19:07:21Z)
• Understanding the Extent to which Summarization Evaluation Metrics Measure the Information Quality of Summaries [74.28810048824519]
  We analyze the token alignments used by ROUGE and BERTScore to compare summaries. We argue that their scores largely cannot be interpreted as measuring information overlap.
  arXiv  Detail & Related papers  (2020-10-23T15:55:15Z)
• Text Mining to Identify and Extract Novel Disease Treatments From Unstructured Datasets [56.38623317907416]
  We use Google Cloud to transcribe podcast episodes of an NPR radio show. We then build a pipeline for systematically pre-processing the text. Our model successfully identified that Omeprazole can help treat heartburn.
  arXiv  Detail & Related papers  (2020-10-22T19:52:49Z)
• Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization [22.611879349101596]
  We evaluate modern neural models for abstractive summarization of relevant article abstracts from systematic reviews. We find that modern summarization systems yield consistently fluent and relevant synopses, but that they are not always factual.
  arXiv  Detail & Related papers  (2020-08-25T22:22:50Z)
• Enhancing Factual Consistency of Abstractive Summarization [57.67609672082137]
  We propose a fact-aware summarization model FASum to extract and integrate factual relations into the summary generation process. We then design a factual corrector model FC to automatically correct factual errors from summaries generated by existing systems.
  arXiv  Detail & Related papers  (2020-03-19T07:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.