Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection

• URL: http://arxiv.org/abs/2104.09061v1
• Date: Mon, 19 Apr 2021 05:39:24 GMT
• Title: Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection
• Authors: Sihao Chen and Fan Zhang and Kazoo Sone and Dan Roth
• Abstract summary: We study contrast candidate generation and selection as a model-agnostic post-processing technique. We learn a discriminative correction model by generating alternative candidate summaries. This model is then used to select the best candidate as the final output summary.
• Score: 54.38512834521367
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Despite significant progress in neural abstractive summarization, recent studies have shown that the current models are prone to generating summaries that are unfaithful to the original context. To address the issue, we study contrast candidate generation and selection as a model-agnostic post-processing technique to correct the extrinsic hallucinations (i.e. information not present in the source text) in unfaithful summaries. We learn a discriminative correction model by generating alternative candidate summaries where named entities and quantities in the generated summary are replaced with ones with compatible semantic types from the source document. This model is then used to select the best candidate as the final output summary. Our experiments and analysis across a number of neural summarization systems show that our proposed method is effective in identifying and correcting extrinsic hallucinations. We analyze the typical hallucination phenomenon by different types of neural summarization systems, in hope to provide insights for future work on the direction.

Related papers

• Improving Sequence-to-Sequence Models for Abstractive Text Summarization Using Meta Heuristic Approaches [0.0]
  Humans have a unique ability to create abstractions. The use of sequence-to-sequence (seq2seq) models for neural abstractive text summarization has been ascending as far as prevalence. In this article, we aim toward enhancing the present architectures and models for abstractive text summarization.
  arXiv  Detail & Related papers  (2024-03-24T17:39:36Z)
• Seeing Unseen: Discover Novel Biomedical Concepts via Geometry-Constrained Probabilistic Modeling [53.7117640028211]
  We present a geometry-constrained probabilistic modeling treatment to resolve the identified issues. We incorporate a suite of critical geometric properties to impose proper constraints on the layout of constructed embedding space. A spectral graph-theoretic method is devised to estimate the number of potential novel classes.
  arXiv  Detail & Related papers  (2024-03-02T00:56:05Z)
• Dual-View Data Hallucination with Semantic Relation Guidance for Few-Shot Image Recognition [49.26065739704278]
  We propose a framework that exploits semantic relations to guide dual-view data hallucination for few-shot image recognition. An instance-view data hallucination module hallucinates each sample of a novel class to generate new data. A prototype-view data hallucination module exploits semantic-aware measure to estimate the prototype of a novel class.
  arXiv  Detail & Related papers  (2024-01-13T12:32:29Z)
• Mutual Information Alleviates Hallucinations in Abstractive Summarization [73.48162198041884]
  We find a simple criterion under which models are significantly more likely to assign more probability to hallucinated content during generation: high model uncertainty. This finding offers a potential explanation for hallucinations: models default to favoring text with high marginal probability, when uncertain about a continuation. We propose a decoding strategy that switches to optimizing for pointwise mutual information of the source and target token--rather than purely the probability of the target token--when the model exhibits uncertainty.
  arXiv  Detail & Related papers  (2022-10-24T13:30:54Z)
• Don't Say What You Don't Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search [54.286450484332505]
  We analyze the connection between hallucinations and training data, and find evidence that models hallucinate because they train on target summaries that are unsupported by the source. We present PINOCCHIO, a new decoding method that improves the consistency of a transformer-based abstractive summarizer by constraining beam search to avoid hallucinations.
  arXiv  Detail & Related papers  (2022-03-16T07:13:52Z)
• Dissecting Generation Modes for Abstractive Summarization Models via Ablation and Attribution [34.2658286826597]
  We propose a two-step method to interpret summarization model decisions. We first analyze the model's behavior by ablating the full model to categorize each decoder decision into one of several generation modes. After isolating decisions that do depend on the input, we explore interpreting these decisions using several different attribution methods.
  arXiv  Detail & Related papers  (2021-06-03T00:54:16Z)
• The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey [25.59111855107199]
  neural encoder-decoder models pioneered by Seq2Seq framework have been proposed to achieve the goal of generating more abstractive summaries. At a high level, such neural models can freely generate summaries without any constraint on the words or phrases used. However, the neural model's abstraction ability is a double-edged sword.
  arXiv  Detail & Related papers  (2021-04-30T08:46:13Z)
• Multi-Fact Correction in Abstractive Text Summarization [98.27031108197944]
  Span-Fact is a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-10-06T02:51:02Z)
• Generating (Factual?) Narrative Summaries of RCTs: Experiments with Neural Multi-Document Summarization [22.611879349101596]
  We evaluate modern neural models for abstractive summarization of relevant article abstracts from systematic reviews. We find that modern summarization systems yield consistently fluent and relevant synopses, but that they are not always factual.
  arXiv  Detail & Related papers  (2020-08-25T22:22:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.