Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality

• URL: http://arxiv.org/abs/2305.14981v1
• Date: Wed, 24 May 2023 10:15:17 GMT
• Title: Improving Factuality of Abstractive Summarization without Sacrificing Summary Quality
• Authors: Tanay Dixit, Fei Wang, Muhao Chen
• Abstract summary: We propose EFACTSUM (i.e., Effective Factual Summarization) to improve summary factuality without sacrificing summary quality. We show that using a contrastive learning framework with our refined candidate summaries leads to significant gains on both factuality and similarity-based metrics.
• Score: 27.57037141986362
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Improving factual consistency of abstractive summarization has been a widely studied topic. However, most of the prior works on training factuality-aware models have ignored the negative effect it has on summary quality. We propose EFACTSUM (i.e., Effective Factual Summarization), a candidate summary generation and ranking technique to improve summary factuality without sacrificing summary quality. We show that using a contrastive learning framework with our refined candidate summaries leads to significant gains on both factuality and similarity-based metrics. Specifically, we propose a ranking strategy in which we effectively combine two metrics, thereby preventing any conflict during training. Models trained using our approach show up to 6 points of absolute improvement over the base model with respect to FactCC on XSUM and 11 points on CNN/DM, without negatively affecting either similarity-based metrics or absractiveness.

Related papers

• Information-Theoretic Distillation for Reference-less Summarization [67.51150817011617]
  We present a novel framework to distill a powerful summarizer based on the information-theoretic objective for summarization. We start off from Pythia-2.8B as the teacher model, which is not yet capable of summarization. We arrive at a compact but powerful summarizer with only 568M parameters that performs competitively against ChatGPT.
  arXiv  Detail & Related papers  (2024-03-20T17:42:08Z)
• Improving Factuality of Abstractive Summarization via Contrastive Reward Learning [77.07192378869776]
  We propose a simple but effective contrastive learning framework that incorporates recent developments in reward learning and factuality metrics. Empirical studies demonstrate that the proposed framework enables summarization models to learn from feedback of factuality metrics.
  arXiv  Detail & Related papers  (2023-07-10T12:01:18Z)
• Curriculum-Guided Abstractive Summarization [45.57561926145256]
  Recent Transformer-based summarization models have provided a promising approach to abstractive summarization. These models have two shortcomings: (1) they often perform poorly in content selection, and (2) their training strategy is not quite efficient, which restricts model performance. In this paper, we explore two ways to compensate for these pitfalls. First, we augment the Transformer network with a sentence cross-attention module in the decoder, encouraging more abstraction of salient content.
  arXiv  Detail & Related papers  (2023-02-02T11:09:37Z)
• Inverse Reinforcement Learning for Text Summarization [52.765898203824975]
  We introduce inverse reinforcement learning (IRL) as an effective paradigm for training abstractive summarization models. Experimental results across datasets in different domains demonstrate the superiority of our proposed IRL model for summarization over MLE and RL baselines.
  arXiv  Detail & Related papers  (2022-12-19T23:45:05Z)
• Evaluating and Improving Factuality in Multimodal Abstractive Summarization [91.46015013816083]
  We propose CLIPBERTScore to leverage the robustness and strong factuality detection performance between image-summary and document-summary. We show that this simple combination of two metrics in the zero-shot achieves higher correlations than existing factuality metrics for document summarization. Our analysis demonstrates the robustness and high correlation of CLIPBERTScore and its components on four factuality metric-evaluation benchmarks.
  arXiv  Detail & Related papers  (2022-11-04T16:50:40Z)
• FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization [91.46015013816083]
  We present FactPEG, an abstractive summarization model that addresses the problem of factuality during pre-training and fine-tuning. Our analysis suggests FactPEG is more factual than using the original pre-training objective in zero-shot and fewshot settings.
  arXiv  Detail & Related papers  (2022-05-16T17:39:14Z)
• CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization [6.017006996402699]
  We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive training data, and automatically generated erroneous summaries, as negative training data, to train summarization systems that are better at distinguishing between them.
  arXiv  Detail & Related papers  (2021-09-19T20:05:21Z)
• Improving Factual Consistency of Abstractive Summarization via Question Answering [25.725873545789046]
  We present an approach to address factual consistency in summarization. We first propose an efficient automatic evaluation metric to measure factual consistency. We then propose a novel learning algorithm that maximizes the proposed metric during model training.
  arXiv  Detail & Related papers  (2021-05-10T19:07:21Z)
• What Have We Achieved on Text Summarization? [32.90169694110989]
  We aim to gain more understanding of summarization systems with respect to their strengths and limits on a fine-grained syntactic and semantic level. We quantify 8 major sources of errors on 10 representative summarization models manually.
  arXiv  Detail & Related papers  (2020-10-09T12:39:33Z)
• Unsupervised Reference-Free Summary Quality Evaluation via Contrastive Learning [66.30909748400023]
  We propose to evaluate the summary qualities without reference summaries by unsupervised contrastive learning. Specifically, we design a new metric which covers both linguistic qualities and semantic informativeness based on BERT. Experiments on Newsroom and CNN/Daily Mail demonstrate that our new evaluation method outperforms other metrics even without reference summaries.
  arXiv  Detail & Related papers  (2020-10-05T05:04:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.