Towards Abstractive Timeline Summarisation using Preference-based Reinforcement Learning

• URL: http://arxiv.org/abs/2211.07596v2
• Date: Thu, 2 Nov 2023 21:16:02 GMT
• Title: Towards Abstractive Timeline Summarisation using Preference-based Reinforcement Learning
• Authors: Yuxuan Ye and Edwin Simpson
• Abstract summary: This paper introduces a novel pipeline for summarising timelines of events reported by multiple news sources. Transformer-based models for abstractive summarisation generate coherent and concise summaries of long documents. While extractive summaries are more faithful to their sources, they may be less readable and contain redundant or unnecessary information.
• Score: 3.6640004265358477
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: This paper introduces a novel pipeline for summarising timelines of events reported by multiple news sources. Transformer-based models for abstractive summarisation generate coherent and concise summaries of long documents but can fail to outperform established extractive methods on specialised tasks such as timeline summarisation (TLS). While extractive summaries are more faithful to their sources, they may be less readable and contain redundant or unnecessary information. This paper proposes a preference-based reinforcement learning (PBRL) method for adapting pretrained abstractive summarisers to TLS, which can overcome the drawbacks of extractive timeline summaries. We define a compound reward function that learns from keywords of interest and pairwise preference labels, which we use to fine-tune a pretrained abstractive summariser via offline reinforcement learning. We carry out both automated and human evaluation on three datasets, finding that our method outperforms a comparable extractive TLS method on two of the three benchmark datasets, and participants prefer our method's summaries to those of both the extractive TLS method and the pretrained abstractive model. The method does not require expensive reference summaries and needs only a small number of preferences to align the generated summaries with human preferences.

Related papers

• Towards Enhancing Coherence in Extractive Summarization: Dataset and Experiments with LLMs [70.15262704746378]
  We propose a systematically created human-annotated dataset consisting of coherent summaries for five publicly available datasets and natural language user feedback. Preliminary experiments with Falcon-40B and Llama-2-13B show significant performance improvements (10% Rouge-L) in terms of producing coherent summaries.
  arXiv  Detail & Related papers  (2024-07-05T20:25:04Z)
• Source Identification in Abstractive Summarization [0.8883733362171033]
  We define input sentences that contain essential information in the generated summary as $textitsource sentences$ and study how abstractive summaries are made by analyzing the source sentences. We formulate automatic source sentence detection and compare multiple methods to establish a strong baseline for the task. Experimental results show that the perplexity-based method performs well in highly abstractive settings, while similarity-based methods robustly in relatively extractive settings.
  arXiv  Detail & Related papers  (2024-02-07T09:09:09Z)
• Salience Allocation as Guidance for Abstractive Summarization [61.31826412150143]
  We propose a novel summarization approach with a flexible and reliable salience guidance, namely SEASON (SaliencE Allocation as Guidance for Abstractive SummarizatiON) SEASON utilizes the allocation of salience expectation to guide abstractive summarization and adapts well to articles in different abstractiveness.
  arXiv  Detail & Related papers  (2022-10-22T02:13:44Z)
• Towards Realistic Low-resource Relation Extraction: A Benchmark with Empirical Baseline Study [51.33182775762785]
  This paper presents an empirical study to build relation extraction systems in low-resource settings. We investigate three schemes to evaluate the performance in low-resource settings: (i) different types of prompt-based methods with few-shot labeled data; (ii) diverse balancing methods to address the long-tailed distribution issue; and (iii) data augmentation technologies and self-training to generate more labeled in-domain data.
  arXiv  Detail & Related papers  (2022-10-19T15:46:37Z)
• Improving Multi-Document Summarization through Referenced Flexible Extraction with Credit-Awareness [21.037841262371355]
  A notable challenge in Multi-Document Summarization (MDS) is the extremely-long length of the input. We present an extract-then-abstract Transformer framework to overcome the problem. We propose a loss weighting mechanism that makes the model aware of the unequal importance for the sentences not in the pseudo extraction oracle.
  arXiv  Detail & Related papers  (2022-05-04T04:40:39Z)
• OTExtSum: Extractive Text Summarisation with Optimal Transport [45.78604902572955]
  We propose a novel non-learning-based method by for the first time formulating text summarisation as an Optimal Transport (OT) problem. Our proposed method outperforms the state-of-the-art non-learning-based methods and several recent learning-based methods in terms of the ROUGE metric.
  arXiv  Detail & Related papers  (2022-04-21T13:25:34Z)
• A New Sentence Extraction Strategy for Unsupervised Extractive Summarization Methods [26.326800624948344]
  We model the task of extractive text summarization methods from the perspective of Information Theory. To improve the feature distribution and to decrease the mutual information of summarization sentences, we propose a new sentence extraction strategy.
  arXiv  Detail & Related papers  (2021-12-06T18:00:02Z)
• The Summary Loop: Learning to Write Abstractive Summaries Without Examples [21.85348918324668]
  This work presents a new approach to unsupervised abstractive summarization based on maximizing a combination of coverage and fluency for a given length constraint. Key terms are masked out of the original document and must be filled in by a coverage model using the current generated summary. When tested on popular news summarization datasets, the method outperforms previous unsupervised methods by more than 2 R-1 points.
  arXiv  Detail & Related papers  (2021-05-11T23:19:46Z)
• Multi-Fact Correction in Abstractive Text Summarization [98.27031108197944]
  Span-Fact is a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-10-06T02:51:02Z)
• SummPip: Unsupervised Multi-Document Summarization with Sentence Graph Compression [61.97200991151141]
  SummPip is an unsupervised method for multi-document summarization. We convert the original documents to a sentence graph, taking both linguistic and deep representation into account. We then apply spectral clustering to obtain multiple clusters of sentences, and finally compress each cluster to generate the final summary.
  arXiv  Detail & Related papers  (2020-07-17T13:01:15Z)
• Pre-training for Abstractive Document Summarization by Reinstating Source Text [105.77348528847337]
  This paper presents three pre-training objectives which allow us to pre-train a Seq2Seq based abstractive summarization model on unlabeled text. Experiments on two benchmark summarization datasets show that all three objectives can improve performance upon baselines.
  arXiv  Detail & Related papers  (2020-04-04T05:06:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.