Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization

• URL: http://arxiv.org/abs/2205.14521v1
• Date: Sat, 28 May 2022 21:09:23 GMT
• Title: Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization
• Authors: Puyuan Liu, Chenyang Huang, Lili Mou
• Abstract summary: Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization approach. Experiments show that NAUS achieves state-of-the-art performance for unsupervised summarization.
• Score: 20.87460375478907
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization (NAUS) approach, which does not require parallel data for training. Our NAUS first performs edit-based search towards a heuristically defined score, and generates a summary as pseudo-groundtruth. Then, we train an encoder-only non-autoregressive Transformer based on the search result. We also propose a dynamic programming approach for length-control decoding, which is important for the summarization task. Experiments on two datasets show that NAUS achieves state-of-the-art performance for unsupervised summarization, yet largely improving inference efficiency. Further, our algorithm is able to perform explicit length-transfer summary generation.

Related papers

• Unsupervised Extractive Summarization with Learnable Length Control Strategies [33.75745103050596]
  Unsupervised extractive summarization is an important technique in information extraction and retrieval. Most of existing unsupervised methods rely on graph-based ranking on sentence centrality. This paper introduces an unsupervised extractive summarization model based on a siamese network.
  arXiv  Detail & Related papers  (2023-12-12T00:15:26Z)
• Contrastive Transformer Learning with Proximity Data Generation for Text-Based Person Search [60.626459715780605]
  Given a descriptive text query, text-based person search aims to retrieve the best-matched target person from an image gallery. Such a cross-modal retrieval task is quite challenging due to significant modality gap, fine-grained differences and insufficiency of annotated data. In this paper, we propose a simple yet effective dual Transformer model for text-based person search.
  arXiv  Detail & Related papers  (2023-11-15T16:26:49Z)
• Summarization Programs: Interpretable Abstractive Summarization with Neural Modular Trees [89.60269205320431]
  Current abstractive summarization models either suffer from a lack of clear interpretability or provide incomplete rationales. We propose the Summarization Program (SP), an interpretable modular framework consisting of an (ordered) list of binary trees. A Summarization Program contains one root node per summary sentence, and a distinct tree connects each summary sentence to the document sentences.
  arXiv  Detail & Related papers  (2022-09-21T16:50:22Z)
• A Character-Level Length-Control Algorithm for Non-Autoregressive Sentence Summarization [23.495225374478295]
  Sentence summarization aims at compressing a long sentence into a short one that keeps the main gist, and has extensive real-world applications such as headline generation. In our work, we address a new problem of explicit character-level length control for summarization, and propose a dynamic programming algorithm based on the Connectionist Temporal Classification (CTC) model.
  arXiv  Detail & Related papers  (2022-05-28T21:09:53Z)
• HETFORMER: Heterogeneous Transformer with Sparse Attention for Long-Text Extractive Summarization [57.798070356553936]
  HETFORMER is a Transformer-based pre-trained model with multi-granularity sparse attentions for extractive summarization. Experiments on both single- and multi-document summarization tasks show that HETFORMER achieves state-of-the-art performance in Rouge F1.
  arXiv  Detail & Related papers  (2021-10-12T22:42:31Z)
• The Summary Loop: Learning to Write Abstractive Summaries Without Examples [21.85348918324668]
  This work presents a new approach to unsupervised abstractive summarization based on maximizing a combination of coverage and fluency for a given length constraint. Key terms are masked out of the original document and must be filled in by a coverage model using the current generated summary. When tested on popular news summarization datasets, the method outperforms previous unsupervised methods by more than 2 R-1 points.
  arXiv  Detail & Related papers  (2021-05-11T23:19:46Z)
• Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers [107.12125265675483]
  Unsupervised extractive document summarization aims to select important sentences from a document without using labeled summaries during training. Existing methods are mostly graph-based with sentences as nodes and edge weights measured by sentence similarities. We find that transformer attentions can be used to rank sentences for unsupervised extractive summarization.
  arXiv  Detail & Related papers  (2020-10-16T08:44:09Z)
• Discrete Optimization for Unsupervised Sentence Summarization with Word-Level Extraction [31.648764677078837]
  Automatic sentence summarization produces a shorter version of a sentence, while preserving its most important information. We model these two aspects in an unsupervised objective function, consisting of language modeling and semantic similarity metrics. Our proposed method achieves a new state-of-the art for unsupervised sentence summarization according to ROUGE scores.
  arXiv  Detail & Related papers  (2020-05-04T19:01:55Z)
• POINTER: Constrained Progressive Text Generation via Insertion-based Generative Pre-training [93.79766670391618]
  We present POINTER, a novel insertion-based approach for hard-constrained text generation. The proposed method operates by progressively inserting new tokens between existing tokens in a parallel manner. The resulting coarse-to-fine hierarchy makes the generation process intuitive and interpretable.
  arXiv  Detail & Related papers  (2020-05-01T18:11:54Z)
• Pre-training for Abstractive Document Summarization by Reinstating Source Text [105.77348528847337]
  This paper presents three pre-training objectives which allow us to pre-train a Seq2Seq based abstractive summarization model on unlabeled text. Experiments on two benchmark summarization datasets show that all three objectives can improve performance upon baselines.
  arXiv  Detail & Related papers  (2020-04-04T05:06:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.