Improving Factual Consistency of Abstractive Summarization via Question Answering

• URL: http://arxiv.org/abs/2105.04623v1
• Date: Mon, 10 May 2021 19:07:21 GMT
• Title: Improving Factual Consistency of Abstractive Summarization via Question Answering
• Authors: Feng Nan, Cicero Nogueira dos Santos, Henghui Zhu, Patrick Ng, Kathleen McKeown, Ramesh Nallapati, Dejiao Zhang, Zhiguo Wang, Andrew O. Arnold, Bing Xiang
• Abstract summary: We present an approach to address factual consistency in summarization. We first propose an efficient automatic evaluation metric to measure factual consistency. We then propose a novel learning algorithm that maximizes the proposed metric during model training.
• Score: 25.725873545789046
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: A commonly observed problem with the state-of-the art abstractive summarization models is that the generated summaries can be factually inconsistent with the input documents. The fact that automatic summarization may produce plausible-sounding yet inaccurate summaries is a major concern that limits its wide application. In this paper we present an approach to address factual consistency in summarization. We first propose an efficient automatic evaluation metric to measure factual consistency; next, we propose a novel learning algorithm that maximizes the proposed metric during model training. Through extensive experiments, we confirm that our method is effective in improving factual consistency and even overall quality of the summaries, as judged by both automatic metrics and human evaluation.

Related papers

• Improving Factuality of Abstractive Summarization via Contrastive Reward Learning [77.07192378869776]
  We propose a simple but effective contrastive learning framework that incorporates recent developments in reward learning and factuality metrics. Empirical studies demonstrate that the proposed framework enables summarization models to learn from feedback of factuality metrics.
  arXiv  Detail & Related papers  (2023-07-10T12:01:18Z)
• Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback [57.816210168909286]
  We leverage recent progress on textual entailment models to address this problem for abstractive summarization systems. We use reinforcement learning with reference-free, textual entailment rewards to optimize for factual consistency. Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience, and conciseness of the generated summaries.
  arXiv  Detail & Related papers  (2023-05-31T21:04:04Z)
• Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality [27.57037141986362]
  We propose EFACTSUM (i.e., Effective Factual Summarization) to improve summary factuality without sacrificing summary quality. We show that using a contrastive learning framework with our refined candidate summaries leads to significant gains on both factuality and similarity-based metrics.
  arXiv  Detail & Related papers  (2023-05-24T10:15:17Z)
• Improving Faithfulness of Abstractive Summarization by Controlling Confounding Effect of Irrelevant Sentences [38.919090721583075]
  We show that factual inconsistency can be caused by irrelevant parts of the input text, which act as confounders. We design a simple multi-task model to control such confounding by leveraging human-annotated relevant sentences when available. Our approach improves faithfulness scores by 20% over strong baselines on AnswerSumm citepfabbri 2021answersumm dataset.
  arXiv  Detail & Related papers  (2022-12-19T18:51:06Z)
• Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infilling [56.70682379371534]
  We show that our approach vastly outperforms prior methods in correcting erroneous summaries. Our model -- FactEdit -- improves factuality scores by over 11 points on CNN/DM and over 31 points on XSum.
  arXiv  Detail & Related papers  (2022-10-22T07:16:19Z)
• Analyzing and Evaluating Faithfulness in Dialogue Summarization [67.07947198421421]
  We first perform the fine-grained human analysis on the faithfulness of dialogue summaries and observe that over 35% of generated summaries are faithfully inconsistent respective the source dialogues. We present a new model-level faithfulness evaluation method. It examines generation models with multi-choice questions created by rule-based transformations.
  arXiv  Detail & Related papers  (2022-10-21T07:22:43Z)
• Evaluation of Automatic Text Summarization using Synthetic Facts [0.0]
  We propose a new automatic reference-less text summarization evaluation system. Our evaluation system is the first system that measures the overarching quality of the text summarization models.
  arXiv  Detail & Related papers  (2022-04-11T05:10:37Z)
• Factual Consistency Evaluation for Text Summarization via Counterfactual Estimation [42.63902468258758]
  We propose a novel metric to evaluate the factual consistency in text summarization via counterfactual estimation. We conduct a series of experiments on three public abstractive text summarization datasets.
  arXiv  Detail & Related papers  (2021-08-30T11:48:41Z)
• Multi-Fact Correction in Abstractive Text Summarization [98.27031108197944]
  Span-Fact is a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-10-06T02:51:02Z)
• Enhancing Factual Consistency of Abstractive Summarization [57.67609672082137]
  We propose a fact-aware summarization model FASum to extract and integrate factual relations into the summary generation process. We then design a factual corrector model FC to automatically correct factual errors from summaries generated by existing systems.
  arXiv  Detail & Related papers  (2020-03-19T07:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.