FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for
Abstractive Summarization
- URL: http://arxiv.org/abs/2205.07830v1
- Date: Mon, 16 May 2022 17:39:14 GMT
- Title: FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for
Abstractive Summarization
- Authors: David Wan, Mohit Bansal
- Abstract summary: We present FactPEG, an abstractive summarization model that addresses the problem of factuality during pre-training and fine-tuning.
Our analysis suggests FactPEG is more factual than using the original pre-training objective in zero-shot and fewshot settings.
- Score: 91.46015013816083
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We present FactPEGASUS, an abstractive summarization model that addresses the
problem of factuality during pre-training and fine-tuning: (1) We augment the
sentence selection strategy of PEGASUS's (Zhang et al., 2020) pre-training
objective to create pseudo-summaries that are both important and factual; (2)
We introduce three complementary components for fine-tuning. The corrector
removes hallucinations present in the reference summary, the contrastor uses
contrastive learning to better differentiate nonfactual summaries from factual
ones, and the connector bridges the gap between the pre-training and
fine-tuning for better transfer of knowledge. Experiments on three downstream
tasks demonstrate that FactPEGASUS substantially improves factuality evaluated
by multiple automatic metrics and humans. Our thorough analysis suggests that
FactPEGASUS is more factual than using the original pre-training objective in
zero-shot and few-shot settings, retains factual behavior more robustly than
strong baselines, and does not rely entirely on becoming more extractive to
improve factuality. Our code and data are publicly available at:
https://github.com/meetdavidwan/factpegasus
Related papers
- Entity-level Factual Adaptiveness of Fine-tuning based Abstractive
Summarization Models [31.84120883461332]
We analyze the robustness of fine-tuning based summarization models to the knowledge conflict.
We introduce a controllable counterfactual data augmentation method.
arXiv Detail & Related papers (2024-02-23T07:53:39Z) - Disentangled Representation Learning with Transmitted Information Bottleneck [57.22757813140418]
We present textbfDisTIB (textbfTransmitted textbfInformation textbfBottleneck for textbfDisd representation learning), a novel objective that navigates the balance between information compression and preservation.
arXiv Detail & Related papers (2023-11-03T03:18:40Z) - Improving Factuality of Abstractive Summarization without Sacrificing
Summary Quality [27.57037141986362]
We propose EFACTSUM (i.e., Effective Factual Summarization) to improve summary factuality without sacrificing summary quality.
We show that using a contrastive learning framework with our refined candidate summaries leads to significant gains on both factuality and similarity-based metrics.
arXiv Detail & Related papers (2023-05-24T10:15:17Z) - Preserving Knowledge Invariance: Rethinking Robustness Evaluation of
Open Information Extraction [50.62245481416744]
We present the first benchmark that simulates the evaluation of open information extraction models in the real world.
We design and annotate a large-scale testbed in which each example is a knowledge-invariant clique.
By further elaborating the robustness metric, a model is judged to be robust if its performance is consistently accurate on the overall cliques.
arXiv Detail & Related papers (2023-05-23T12:05:09Z) - A Pretrainer's Guide to Training Data: Measuring the Effects of Data
Age, Domain Coverage, Quality, & Toxicity [84.6421260559093]
This study is the largest set of experiments to validate, quantify, and expose undocumented intuitions about text pretraining.
Our findings indicate there does not exist a one-size-fits-all solution to filtering training data.
arXiv Detail & Related papers (2023-05-22T15:57:53Z) - Correcting Diverse Factual Errors in Abstractive Summarization via
Post-Editing and Language Model Infilling [56.70682379371534]
We show that our approach vastly outperforms prior methods in correcting erroneous summaries.
Our model -- FactEdit -- improves factuality scores by over 11 points on CNN/DM and over 31 points on XSum.
arXiv Detail & Related papers (2022-10-22T07:16:19Z) - Counterfactual Data Augmentation improves Factuality of Abstractive
Summarization [6.745946263790011]
We show that augmenting the training data with our approach improves the factual correctness of summaries without significantly affecting the ROUGE score.
We show that in two commonly used summarization datasets (CNN/Dailymail and XSum), we improve the factual correctness by about 2.5 points on average.
arXiv Detail & Related papers (2022-05-25T00:00:35Z) - CO2Sum:Contrastive Learning for Factual-Consistent Abstractive
Summarization [11.033949886530225]
CO2Sum is a contrastive learning scheme that can be easily applied on sequence-to-sequence models.
Experiments on public benchmarks demonstrate that CO2Sum improves the faithfulness on large pre-trained language models.
arXiv Detail & Related papers (2021-12-02T11:52:01Z) - CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in
Abstractive Summarization [6.017006996402699]
We study generating abstractive summaries that are faithful and factually consistent with the given articles.
A novel contrastive learning formulation is presented, which leverages both reference summaries, as positive training data, and automatically generated erroneous summaries, as negative training data, to train summarization systems that are better at distinguishing between them.
arXiv Detail & Related papers (2021-09-19T20:05:21Z) - Pre-training for Abstractive Document Summarization by Reinstating
Source Text [105.77348528847337]
This paper presents three pre-training objectives which allow us to pre-train a Seq2Seq based abstractive summarization model on unlabeled text.
Experiments on two benchmark summarization datasets show that all three objectives can improve performance upon baselines.
arXiv Detail & Related papers (2020-04-04T05:06:26Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.