Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback

• URL: http://arxiv.org/abs/2306.00186v1
• Date: Wed, 31 May 2023 21:04:04 GMT
• Title: Factually Consistent Summarization via Reinforcement Learning with Textual Entailment Feedback
• Authors: Paul Roit, Johan Ferret, Lior Shani, Roee Aharoni, Geoffrey Cideron, Robert Dadashi, Matthieu Geist, Sertan Girgin, L\'eonard Hussenot, Orgad Keller, Nikola Momchev, Sabela Ramos, Piotr Stanczyk, Nino Vieillard, Olivier Bachem, Gal Elidan, Avinatan Hassidim, Olivier Pietquin and Idan Szpektor
• Abstract summary: We leverage recent progress on textual entailment models to address this problem for abstractive summarization systems. We use reinforcement learning with reference-free, textual entailment rewards to optimize for factual consistency. Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience, and conciseness of the generated summaries.
• Score: 57.816210168909286
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the seeming success of contemporary grounded text generation systems, they often tend to generate factually inconsistent text with respect to their input. This phenomenon is emphasized in tasks like summarization, in which the generated summaries should be corroborated by their source article. In this work, we leverage recent progress on textual entailment models to directly address this problem for abstractive summarization systems. We use reinforcement learning with reference-free, textual entailment rewards to optimize for factual consistency and explore the ensuing trade-offs, as improved consistency may come at the cost of less informative or more extractive summaries. Our results, according to both automatic metrics and human evaluation, show that our method considerably improves the faithfulness, salience, and conciseness of the generated summaries.

Related papers

• Improving Factuality of Abstractive Summarization via Contrastive Reward Learning [77.07192378869776]
  We propose a simple but effective contrastive learning framework that incorporates recent developments in reward learning and factuality metrics. Empirical studies demonstrate that the proposed framework enables summarization models to learn from feedback of factuality metrics.
  arXiv  Detail & Related papers  (2023-07-10T12:01:18Z)
• SummIt: Iterative Text Summarization via ChatGPT [12.966825834765814]
  We propose SummIt, an iterative text summarization framework based on large language models like ChatGPT. Our framework enables the model to refine the generated summary iteratively through self-evaluation and feedback. We also conduct a human evaluation to validate the effectiveness of the iterative refinements and identify a potential issue of over-correction.
  arXiv  Detail & Related papers  (2023-05-24T07:40:06Z)
• The Exploration of Knowledge-Preserving Prompts for Document Summarisation [9.580602790508141]
  This study explores the possibility of adopting prompts to incorporate factual knowledge into generated summaries. Experimental results demonstrate that the trainable prefixes can help the summarisation model extract information from discrete prompts precisely. The ROUGE improvements of the generated summaries indicate that explicitly adding factual knowledge into the summarisation process could boost the overall performance.
  arXiv  Detail & Related papers  (2023-01-27T14:05:12Z)
• Generating Multiple-Length Summaries via Reinforcement Learning for Unsupervised Sentence Summarization [44.835811239393244]
  Sentence summarization shortens given texts while maintaining core contents of the texts. Unsupervised approaches have been studied to summarize texts without human-written summaries. We devise an abstractive model based on reinforcement learning without ground-truth summaries.
  arXiv  Detail & Related papers  (2022-12-21T08:34:28Z)
• Analyzing and Evaluating Faithfulness in Dialogue Summarization [67.07947198421421]
  We first perform the fine-grained human analysis on the faithfulness of dialogue summaries and observe that over 35% of generated summaries are faithfully inconsistent respective the source dialogues. We present a new model-level faithfulness evaluation method. It examines generation models with multi-choice questions created by rule-based transformations.
  arXiv  Detail & Related papers  (2022-10-21T07:22:43Z)
• Contextualized Rewriting for Text Summarization [10.666547385992935]
  We formalized rewriting as a seq2seq problem with group alignments. Results show that our approach significantly outperforms non-contextualized rewriting systems.
  arXiv  Detail & Related papers  (2021-01-31T05:35:57Z)
• Multi-Fact Correction in Abstractive Text Summarization [98.27031108197944]
  Span-Fact is a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-10-06T02:51:02Z)
• Few-Shot Learning for Opinion Summarization [117.70510762845338]
  Opinion summarization is the automatic creation of text reflecting subjective information expressed in multiple documents. In this work, we show that even a handful of summaries is sufficient to bootstrap generation of the summary text. Our approach substantially outperforms previous extractive and abstractive methods in automatic and human evaluation.
  arXiv  Detail & Related papers  (2020-04-30T15:37:38Z)
• Enhancing Factual Consistency of Abstractive Summarization [57.67609672082137]
  We propose a fact-aware summarization model FASum to extract and integrate factual relations into the summary generation process. We then design a factual corrector model FC to automatically correct factual errors from summaries generated by existing systems.
  arXiv  Detail & Related papers  (2020-03-19T07:36:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.