Improving the Faithfulness of Abstractive Summarization via Entity Coverage Control

• URL: http://arxiv.org/abs/2207.02263v1
• Date: Tue, 5 Jul 2022 18:52:19 GMT
• Title: Improving the Faithfulness of Abstractive Summarization via Entity Coverage Control
• Authors: Haopeng Zhang, Semih Yavuz, Wojciech Kryscinski, Kazuma Hashimoto, Yingbo Zhou
• Abstract summary: We propose a method to remedy entity-level hallucinations with Entity Coverage Control (ECC) ECC computes entity coverage precision and prepend the corresponding control code for each training example. We show that the proposed method leads to more faithful and salient abstractive summarization in supervised fine-tuning and zero-shot settings.
• Score: 27.214742188672464
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Abstractive summarization systems leveraging pre-training language models have achieved superior results on benchmark datasets. However, such models have been shown to be more prone to hallucinate facts that are unfaithful to the input context. In this paper, we propose a method to remedy entity-level extrinsic hallucinations with Entity Coverage Control (ECC). We first compute entity coverage precision and prepend the corresponding control code for each training example, which implicitly guides the model to recognize faithfulness contents in the training phase. We further extend our method via intermediate fine-tuning on large but noisy data extracted from Wikipedia to unlock zero-shot summarization. We show that the proposed method leads to more faithful and salient abstractive summarization in supervised fine-tuning and zero-shot settings according to our experimental results on three benchmark datasets XSum, Pubmed, and SAMSum of very different domains and styles.

Related papers

• Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs [70.15262704746378]
  We propose a systematically created human-annotated dataset consisting of coherent summaries for five publicly available datasets and natural language user feedback. Preliminary experiments with Falcon-40B and Llama-2-13B show significant performance improvements (10% Rouge-L) in terms of producing coherent summaries.
  arXiv  Detail & Related papers  (2024-07-05T20:25:04Z)
• Semi-Supervised Dialogue Abstractive Summarization via High-Quality Pseudolabel Selection [27.531083525683243]
  Semi-supervised dialogue summarization (SSDS) leverages model-generated summaries to reduce reliance on human-labeled data. We propose a novel scoring approach, SiCF, which encapsulates three primary dimensions of summarization model quality.
  arXiv  Detail & Related papers  (2024-03-06T22:06:23Z)
• Towards Faithful Explanations for Text Classification with Robustness Improvement and Explanation Guided Training [30.626080706755822]
  Feature attribution methods highlight the important input tokens as explanations to model predictions. Recent works show that explanations provided by these methods face challenges of being faithful and robust. We propose a method with Robustness improvement and Explanation Guided training towards more faithful EXplanations (REGEX) for text classification.
  arXiv  Detail & Related papers  (2023-12-29T13:07:07Z)
• AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation [57.8363998797433]
  We propose AMRFact, a framework that generates perturbed summaries using Abstract Meaning Representations (AMRs) Our approach parses factually consistent summaries into AMR graphs and injects controlled factual inconsistencies to create negative examples, allowing for coherent factually inconsistent summaries to be generated with high error-type coverage.
  arXiv  Detail & Related papers  (2023-11-16T02:56:29Z)
• COLO: A Contrastive Learning based Re-ranking Framework for One-Stage Summarization [84.70895015194188]
  We propose a Contrastive Learning based re-ranking framework for one-stage summarization called COLO. COLO boosts the extractive and abstractive results of one-stage systems on CNN/DailyMail benchmark to 44.58 and 46.33 ROUGE-1 score.
  arXiv  Detail & Related papers  (2022-09-29T06:11:21Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)
• Unsupervised Opinion Summarization with Content Planning [58.5308638148329]
  We show that explicitly incorporating content planning in a summarization model yields output of higher quality. We also create synthetic datasets which are more natural, resembling real world document-summary pairs. Our approach outperforms competitive models in generating informative, coherent, and fluent summaries.
  arXiv  Detail & Related papers  (2020-12-14T18:41:58Z)
• Improving Zero and Few-Shot Abstractive Summarization with Intermediate Fine-tuning and Data Augmentation [101.26235068460551]
  Models pretrained with self-supervised objectives on large text corpora achieve state-of-the-art performance on English text summarization tasks. Models are typically fine-tuned on hundreds of thousands of data points, an infeasible requirement when applying summarization to new, niche domains. We introduce a novel and generalizable method, called WikiTransfer, for fine-tuning pretrained models for summarization in an unsupervised, dataset-specific manner.
  arXiv  Detail & Related papers  (2020-10-24T08:36:49Z)
• Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation [37.054709598792165]
  The model is a convolutional neural network that operates directly on the raw waveform. It is optimized to identify spectral changes in the signal using the Noise-Contrastive Estimation principle. At test time, a peak detection algorithm is applied over the model outputs to produce the final boundaries.
  arXiv  Detail & Related papers  (2020-07-27T12:10:21Z)
• Self-Supervised and Controlled Multi-Document Opinion Summarization [16.674646504295687]
  We propose a self-supervised setup that considers an individual document as a target summary for a set of similar documents. We address the problem of hallucinations through the use of control codes. Our benchmarks on two datasets against graph-based and recent neural abstractive unsupervised models show that our proposed method generates summaries with a superior quality and relevance.
  arXiv  Detail & Related papers  (2020-04-30T13:20:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.