Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders

• URL: http://arxiv.org/abs/2310.18992v1
• Date: Sun, 29 Oct 2023 12:27:18 GMT
• Title: Bipartite Graph Pre-training for Unsupervised Extractive Summarization with Graph Convolutional Auto-Encoders
• Authors: Qianren Mao and Shaobo Zhao and Jiarui Li and Xiaolei Gu and Shizhu He and Bo Li and Jianxin Li
• Abstract summary: We argue that utilizing pre-trained embeddings derived from a process specifically designed to optimize cohensive and distinctive sentence representations helps rank significant sentences. We propose a novel graph pre-training auto-encoder to obtain sentence embeddings by explicitly modelling intra-sentential distinctive features and inter-sentential cohesive features.
• Score: 24.13261636386226
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Pre-trained sentence representations are crucial for identifying significant sentences in unsupervised document extractive summarization. However, the traditional two-step paradigm of pre-training and sentence-ranking, creates a gap due to differing optimization objectives. To address this issue, we argue that utilizing pre-trained embeddings derived from a process specifically designed to optimize cohensive and distinctive sentence representations helps rank significant sentences. To do so, we propose a novel graph pre-training auto-encoder to obtain sentence embeddings by explicitly modelling intra-sentential distinctive features and inter-sentential cohesive features through sentence-word bipartite graphs. These pre-trained sentence representations are then utilized in a graph-based ranking algorithm for unsupervised summarization. Our method produces predominant performance for unsupervised summarization frameworks by providing summary-worthy sentence representations. It surpasses heavy BERT- or RoBERTa-based sentence representations in downstream tasks.

Related papers

• Non-Autoregressive Sentence Ordering [22.45972496989434]
  We propose a novel Non-Autoregressive Ordering Network, dubbed textitNAON, which explores bilateral dependencies between sentences and predicts the sentence for each position in parallel. We conduct extensive experiments on several common-used datasets and the experimental results show that our method outperforms all the autoregressive approaches.
  arXiv  Detail & Related papers  (2023-10-19T10:57:51Z)
• Scientific Paper Extractive Summarization Enhanced by Citation Graphs [50.19266650000948]
  We focus on leveraging citation graphs to improve scientific paper extractive summarization under different settings. Preliminary results demonstrate that citation graph is helpful even in a simple unsupervised framework. Motivated by this, we propose a Graph-based Supervised Summarization model (GSS) to achieve more accurate results on the task when large-scale labeled data are available.
  arXiv  Detail & Related papers  (2022-12-08T11:53:12Z)
• Unsupervised Extractive Summarization with Heterogeneous Graph Embeddings for Chinese Document [5.9630342951482085]
  We propose an unsupervised extractive summarizaiton method with heterogeneous graph embeddings (HGEs) for Chinese document. Experimental results demonstrate that our method consistently outperforms the strong baseline in three summarization datasets.
  arXiv  Detail & Related papers  (2022-11-09T06:07:31Z)
• Learning Non-Autoregressive Models from Search for Unsupervised Sentence Summarization [20.87460375478907]
  Text summarization aims to generate a short summary for an input text. In this work, we propose a Non-Autoregressive Unsupervised Summarization approach. Experiments show that NAUS achieves state-of-the-art performance for unsupervised summarization.
  arXiv  Detail & Related papers  (2022-05-28T21:09:23Z)
• Probing as Quantifying the Inductive Bias of Pre-trained Representations [99.93552997506438]
  We present a novel framework for probing where the goal is to evaluate the inductive bias of representations for a particular task. We apply our framework to a series of token-, arc-, and sentence-level tasks.
  arXiv  Detail & Related papers  (2021-10-15T22:01:16Z)
• Narrative Incoherence Detection [76.43894977558811]
  We propose the task of narrative incoherence detection as a new arena for inter-sentential semantic understanding. Given a multi-sentence narrative, decide whether there exist any semantic discrepancies in the narrative flow.
  arXiv  Detail & Related papers  (2020-12-21T07:18:08Z)
• Unsupervised Extractive Summarization by Pre-training Hierarchical Transformers [107.12125265675483]
  Unsupervised extractive document summarization aims to select important sentences from a document without using labeled summaries during training. Existing methods are mostly graph-based with sentences as nodes and edge weights measured by sentence similarities. We find that transformer attentions can be used to rank sentences for unsupervised extractive summarization.
  arXiv  Detail & Related papers  (2020-10-16T08:44:09Z)
• Discrete Optimization for Unsupervised Sentence Summarization with Word-Level Extraction [31.648764677078837]
  Automatic sentence summarization produces a shorter version of a sentence, while preserving its most important information. We model these two aspects in an unsupervised objective function, consisting of language modeling and semantic similarity metrics. Our proposed method achieves a new state-of-the art for unsupervised sentence summarization according to ROUGE scores.
  arXiv  Detail & Related papers  (2020-05-04T19:01:55Z)
• Structure-Augmented Text Representation Learning for Efficient Knowledge Graph Completion [53.31911669146451]
  Human-curated knowledge graphs provide critical supportive information to various natural language processing tasks. These graphs are usually incomplete, urging auto-completion of them. graph embedding approaches, e.g., TransE, learn structured knowledge via representing graph elements into dense embeddings. textual encoding approaches, e.g., KG-BERT, resort to graph triple's text and triple-level contextualized representations.
  arXiv  Detail & Related papers  (2020-04-30T13:50:34Z)
• Pseudo-Convolutional Policy Gradient for Sequence-to-Sequence Lip-Reading [96.48553941812366]
  Lip-reading aims to infer the speech content from the lip movement sequence. Traditional learning process of seq2seq models suffers from two problems. We propose a novel pseudo-convolutional policy gradient (PCPG) based method to address these two problems.
  arXiv  Detail & Related papers  (2020-03-09T09:12:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.