Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization

• URL: http://arxiv.org/abs/2108.02859v2
• Date: Mon, 24 Apr 2023 20:48:32 GMT
• Title: Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization
• Authors: Markus Dreyer, Mengwen Liu, Feng Nan, Sandeep Atluri, Sujith Ravi
• Abstract summary: We analyze the tradeoff between abstractiveness and factuality of generated summaries across multiple datasets and models. We propose new factuality metrics that adjust for the degree of abstractiveness.
• Score: 20.83986393847262
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural models for abstractive summarization tend to generate output that is fluent and well-formed but lacks semantic faithfulness, or factuality, with respect to the input documents. In this paper, we analyze the tradeoff between abstractiveness and factuality of generated summaries across multiple datasets and models, using extensive human evaluations of factuality. In our analysis, we visualize the rates of change in factuality as we gradually increase abstractiveness using a decoding constraint, and we observe that, while increased abstractiveness generally leads to a drop in factuality, the rate of factuality decay depends on factors such as the data that the system was trained on. We introduce two datasets with human factuality judgements; one containing 10.2k generated summaries with systematically varied degrees of abstractiveness; the other containing 4.2k summaries from five different summarization models. We propose new factuality metrics that adjust for the degree of abstractiveness, and we use them to compare the abstractiveness-adjusted factuality of previous summarization works, providing baselines for future work.

Related papers

• Inducing Causal Structure for Abstractive Text Summarization [76.1000380429553]
  We introduce a Structural Causal Model (SCM) to induce the underlying causal structure of the summarization data. We propose a Causality Inspired Sequence-to-Sequence model (CI-Seq2Seq) to learn the causal representations that can mimic the causal factors. Experimental results on two widely used text summarization datasets demonstrate the advantages of our approach.
  arXiv  Detail & Related papers  (2023-08-24T16:06:36Z)
• Learning with Rejection for Abstractive Text Summarization [42.15551472507393]
  We propose a training objective for abstractive summarization based on rejection learning. We show that our method considerably improves the factuality of generated summaries in automatic and human evaluations.
  arXiv  Detail & Related papers  (2023-02-16T19:07:08Z)
• Salience Allocation as Guidance for Abstractive Summarization [61.31826412150143]
  We propose a novel summarization approach with a flexible and reliable salience guidance, namely SEASON (SaliencE Allocation as Guidance for Abstractive SummarizatiON) SEASON utilizes the allocation of salience expectation to guide abstractive summarization and adapts well to articles in different abstractiveness.
  arXiv  Detail & Related papers  (2022-10-22T02:13:44Z)
• CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization [6.017006996402699]
  We study generating abstractive summaries that are faithful and factually consistent with the given articles. A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive training data, and automatically generated erroneous summaries, as negative training data, to train summarization systems that are better at distinguishing between them.
  arXiv  Detail & Related papers  (2021-09-19T20:05:21Z)
• Subjective Bias in Abstractive Summarization [11.675414451656568]
  We formulate the differences among possible multiple expressions summarizing the same content as subjective bias and examine the role of this bias in the context of abstractive summarization. Results of summarization models trained on style-clustered datasets show that there are certain types of styles that lead to better convergence, abstraction and generalization.
  arXiv  Detail & Related papers  (2021-06-18T12:17:55Z)
• Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation [93.87095877617968]
  We propose Constrained Abstractive Summarization (CAS), a general setup that preserves the factual consistency of abstractive summarization. We adopt lexically constrained decoding, a technique generally applicable to autoregressive generative models, to fulfill CAS. We observe up to 13.8 ROUGE-2 gains when only one manual constraint is used in interactive summarization.
  arXiv  Detail & Related papers  (2020-10-24T00:27:44Z)
• Multi-Fact Correction in Abstractive Text Summarization [98.27031108197944]
  Span-Fact is a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-10-06T02:51:02Z)
• On Faithfulness and Factuality in Abstractive Summarization [17.261247316769484]
  We analyzed limitations of neural text generation models for abstractive document summarization. We found that these models are highly prone to hallucinate content that is unfaithful to the input document. We show that textual entailment measures better correlate with faithfulness than standard metrics.
  arXiv  Detail & Related papers  (2020-05-02T00:09:16Z)
• Unsupervised Opinion Summarization with Noising and Denoising [85.49169453434554]
  We create a synthetic dataset from a corpus of user reviews by sampling a review, pretending it is a summary, and generating noisy versions thereof. At test time, the model accepts genuine reviews and generates a summary containing salient opinions, treating those that do not reach consensus as noise.
  arXiv  Detail & Related papers  (2020-04-21T16:54:57Z)
• Enhancing Factual Consistency of Abstractive Summarization [57.67609672082137]
  We propose a fact-aware summarization model FASum to extract and integrate factual relations into the summary generation process. We then design a factual corrector model FC to automatically correct factual errors from summaries generated by existing systems.
  arXiv  Detail & Related papers  (2020-03-19T07:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.