GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization

• URL: http://arxiv.org/abs/2208.09982v1
• Date: Sun, 21 Aug 2022 23:09:29 GMT
• Title: GRETEL: Graph Contrastive Topic Enhanced Language Model for Long Document Extractive Summarization
• Authors: Qianqian Xie and Jimin Huang and Tulika Saha and Sophia Ananiadou
• Abstract summary: We propose a graph contrastive topic enhanced language model (GRETEL) for capturing global semantic information. GRETEL integrates the hierarchical transformer encoder and the graph contrastive learning to fuse the semantic information from the global document context and the gold summary. Experimental results on both general domain and biomedical datasets demonstrate that our proposed method outperforms SOTA methods.
• Score: 22.053942327552583
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recently, neural topic models (NTMs) have been incorporated into pre-trained language models (PLMs), to capture the global semantic information for text summarization. However, in these methods, there remain limitations in the way they capture and integrate the global semantic information. In this paper, we propose a novel model, the graph contrastive topic enhanced language model (GRETEL), that incorporates the graph contrastive topic model with the pre-trained language model, to fully leverage both the global and local contextual semantics for long document extractive summarization. To better capture and incorporate the global semantic information into PLMs, the graph contrastive topic model integrates the hierarchical transformer encoder and the graph contrastive learning to fuse the semantic information from the global document context and the gold summary. To this end, GRETEL encourages the model to efficiently extract salient sentences that are topically related to the gold summary, rather than redundant sentences that cover sub-optimal topics. Experimental results on both general domain and biomedical datasets demonstrate that our proposed method outperforms SOTA methods.

Related papers

• Bridging Local Details and Global Context in Text-Attributed Graphs [62.522550655068336]
  GraphBridge is a framework that bridges local and global perspectives by leveraging contextual textual information. Our method achieves state-of-theart performance, while our graph-aware token reduction module significantly enhances efficiency and solves scalability issues.
  arXiv  Detail & Related papers  (2024-06-18T13:35:25Z)
• Information-Theoretic Distillation for Reference-less Summarization [67.51150817011617]
  We present a novel framework to distill a powerful summarizer based on the information-theoretic objective for summarization. We start off from Pythia-2.8B as the teacher model, which is not yet capable of summarization. We arrive at a compact but powerful summarizer with only 568M parameters that performs competitively against ChatGPT.
  arXiv  Detail & Related papers  (2024-03-20T17:42:08Z)
• Let the Pretrained Language Models "Imagine" for Short Texts Topic Modeling [29.87929724277381]
  In short texts, co-occurrence information is minimal, which results in feature sparsity in document representation. Existing topic models (probabilistic or neural) mostly fail to mine patterns from them to generate coherent topics. We extend short text into longer sequences using existing pre-trained language models (PLMs)
  arXiv  Detail & Related papers  (2023-10-24T00:23:30Z)
• Scientific Paper Extractive Summarization Enhanced by Citation Graphs [50.19266650000948]
  We focus on leveraging citation graphs to improve scientific paper extractive summarization under different settings. Preliminary results demonstrate that citation graph is helpful even in a simple unsupervised framework. Motivated by this, we propose a Graph-based Supervised Summarization model (GSS) to achieve more accurate results on the task when large-scale labeled data are available.
  arXiv  Detail & Related papers  (2022-12-08T11:53:12Z)
• Enriching and Controlling Global Semantics for Text Summarization [11.037667460077813]
  Transformer-based models have been proven effective in the abstractive summarization task by creating fluent and informative summaries. We introduce a neural topic model empowered with normalizing flow to capture the global semantics of the document, which are then integrated into the summarization model. Our method outperforms state-of-the-art summarization models on five common text summarization datasets.
  arXiv  Detail & Related papers  (2021-09-22T09:31:50Z)
• Dialogue Summarization with Supporting Utterance Flow Modeling and Fact Regularization [58.965859508695225]
  We propose an end-to-end neural model for dialogue summarization with two novel modules. The supporting utterance flow modeling helps to generate a coherent summary by smoothly shifting the focus from the former utterances to the later ones. The fact regularization encourages the generated summary to be factually consistent with the ground-truth summary during model training.
  arXiv  Detail & Related papers  (2021-08-03T03:09:25Z)
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph [53.70520466556453]
  We propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow. In addition, a progressive learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph.
  arXiv  Detail & Related papers  (2021-05-06T12:20:41Z)
• Topic-Guided Abstractive Text Summarization: a Joint Learning Approach [19.623946402970933]
  We introduce a new approach for abstractive text summarization, Topic-Guided Abstractive Summarization. The idea is to incorporate neural topic modeling with a Transformer-based sequence-to-sequence (seq2seq) model in a joint learning framework.
  arXiv  Detail & Related papers  (2020-10-20T14:45:25Z)
• Enhancing Extractive Text Summarization with Topic-Aware Graph Neural Networks [21.379555672973975]
  This paper proposes a graph neural network (GNN)-based extractive summarization model. Our model integrates a joint neural topic model (NTM) to discover latent topics, which can provide document-level features for sentence selection. The experimental results demonstrate that our model achieves substantially state-of-the-art results on CNN/DM and NYT datasets.
  arXiv  Detail & Related papers  (2020-10-13T09:30:04Z)
• Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward [42.925345819778656]
  We present ASGARD, a novel framework for Abstractive Summarization with Graph-Augmentation and semantic-driven RewarD. We propose the use of dual encoders---a sequential document encoder and a graph-structured encoder---to maintain the global context and local characteristics of entities. Results show that our models produce significantly higher ROUGE scores than a variant without knowledge graph as input on both New York Times and CNN/Daily Mail datasets.
  arXiv  Detail & Related papers  (2020-05-03T18:23:06Z)
• Towards Making the Most of Context in Neural Machine Translation [112.9845226123306]
  We argue that previous research did not make a clear use of the global context. We propose a new document-level NMT framework that deliberately models the local context of each sentence.
  arXiv  Detail & Related papers  (2020-02-19T03:30:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.