ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization

• URL: http://arxiv.org/abs/2109.04098v1
• Date: Thu, 9 Sep 2021 08:35:39 GMT
• Title: ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization
• Authors: Alireza Salemi, Emad Kebriaei, Ghazal Neisi Minaei, Azadeh Shakery
• Abstract summary: We propose ARMAN, a Transformer-based encoder-decoder model pre-trained with three novel objectives to address this issue. In ARMAN, salient sentences from a document are selected according to a modified semantic score to be masked and form a pseudo summary. We show that our proposed model achieves state-of-the-art performance on all six summarization tasks measured by ROUGE and BERTScore.
• Score: 7.16879432974126
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Abstractive text summarization is one of the areas influenced by the emergence of pre-trained language models. Current pre-training works in abstractive summarization give more points to the summaries with more words in common with the main text and pay less attention to the semantic similarity between generated sentences and the original document. We propose ARMAN, a Transformer-based encoder-decoder model pre-trained with three novel objectives to address this issue. In ARMAN, salient sentences from a document are selected according to a modified semantic score to be masked and form a pseudo summary. To summarize more accurately and similar to human writing patterns, we applied modified sentence reordering. We evaluated our proposed models on six downstream Persian summarization tasks. Experimental results show that our proposed model achieves state-of-the-art performance on all six summarization tasks measured by ROUGE and BERTScore. Our models also outperform prior works in textual entailment, question paraphrasing, and multiple choice question answering. Finally, we established a human evaluation and show that using the semantic score significantly improves summarization results.

Related papers

• Assessment of Transformer-Based Encoder-Decoder Model for Human-Like Summarization [0.05852077003870416]
  This work leverages transformer-based BART model for human-like summarization. On training and fine-tuning the encoder-decoder model, it is tested with diverse sample articles. The finetuned model performance is compared with the baseline pretrained model. Empirical results on BBC News articles highlight that the gold standard summaries written by humans are more factually consistent by 17%.
  arXiv  Detail & Related papers  (2024-10-22T09:25:04Z)
• Rank Your Summaries: Enhancing Bengali Text Summarization via Ranking-based Approach [0.0]
  This paper aims to identify the most accurate and informative summary for a given text by utilizing a simple but effective ranking-based approach. We utilize four pre-trained summarization models to generate summaries, followed by applying a text ranking algorithm to identify the most suitable summary. Experimental results suggest that by leveraging the strengths of each pre-trained transformer model, our methodology significantly improves the accuracy and effectiveness of the Bengali text summarization.
  arXiv  Detail & Related papers  (2023-07-14T15:07:20Z)
• Speech Summarization using Restricted Self-Attention [79.89680891246827]
  We introduce a single model optimized end-to-end for speech summarization. We demonstrate that the proposed model learns to directly summarize speech for the How-2 corpus of instructional videos.
  arXiv  Detail & Related papers  (2021-10-12T18:21:23Z)
• Fine-tuning GPT-3 for Russian Text Summarization [77.34726150561087]
  This paper showcases ruGPT3 ability to summarize texts, fine-tuning it on the corpora of Russian news with their corresponding human-generated summaries. We evaluate the resulting texts with a set of metrics, showing that our solution can surpass the state-of-the-art model's performance without additional changes in architecture or loss function.
  arXiv  Detail & Related papers  (2021-08-07T19:01:40Z)
• Transductive Learning for Abstractive News Summarization [24.03781438153328]
  We propose the first application of transductive learning to summarization. We show that our approach yields state-of-the-art results on CNN/DM and NYT datasets.
  arXiv  Detail & Related papers  (2021-04-17T17:33:12Z)
• Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers [107.12125265675483]
  Unsupervised extractive document summarization aims to select important sentences from a document without using labeled summaries during training. Existing methods are mostly graph-based with sentences as nodes and edge weights measured by sentence similarities. We find that transformer attentions can be used to rank sentences for unsupervised extractive summarization.
  arXiv  Detail & Related papers  (2020-10-16T08:44:09Z)
• Multi-Fact Correction in Abstractive Text Summarization [98.27031108197944]
  Span-Fact is a suite of two factual correction models that leverages knowledge learned from question answering models to make corrections in system-generated summaries via span selection. Our models employ single or multi-masking strategies to either iteratively or auto-regressively replace entities in order to ensure semantic consistency w.r.t. the source text. Experiments show that our models significantly boost the factual consistency of system-generated summaries without sacrificing summary quality in terms of both automatic metrics and human evaluation.
  arXiv  Detail & Related papers  (2020-10-06T02:51:02Z)
• Extractive Summarization as Text Matching [123.09816729675838]
  This paper creates a paradigm shift with regard to the way we build neural extractive summarization systems. We formulate the extractive summarization task as a semantic text matching problem. We have driven the state-of-the-art extractive result on CNN/DailyMail to a new level (44.41 in ROUGE-1)
  arXiv  Detail & Related papers  (2020-04-19T08:27:57Z)
• Pre-training for Abstractive Document Summarization by Reinstating Source Text [105.77348528847337]
  This paper presents three pre-training objectives which allow us to pre-train a Seq2Seq based abstractive summarization model on unlabeled text. Experiments on two benchmark summarization datasets show that all three objectives can improve performance upon baselines.
  arXiv  Detail & Related papers  (2020-04-04T05:06:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.