Enhancing Keyphrase Extraction from Long Scientific Documents using Graph Embeddings

• URL: http://arxiv.org/abs/2305.09316v1
• Date: Tue, 16 May 2023 09:44:38 GMT
• Title: Enhancing Keyphrase Extraction from Long Scientific Documents using Graph Embeddings
• Authors: Roberto Mart\'inez-Cruz, Debanjan Mahata, Alvaro J.L\'opez-L\'opez, Jos\'e Portela
• Abstract summary: We show that augmenting a language model with graph embeddings provides a more comprehensive semantic understanding of words. We demonstrate that enhancing PLMs with graph embeddings outperforms state-of-the-art models on long documents.
• Score: 9.884735234974967
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this study, we investigate using graph neural network (GNN) representations to enhance contextualized representations of pre-trained language models (PLMs) for keyphrase extraction from lengthy documents. We show that augmenting a PLM with graph embeddings provides a more comprehensive semantic understanding of words in a document, particularly for long documents. We construct a co-occurrence graph of the text and embed it using a graph convolutional network (GCN) trained on the task of edge prediction. We propose a graph-enhanced sequence tagging architecture that augments contextualized PLM embeddings with graph representations. Evaluating on benchmark datasets, we demonstrate that enhancing PLMs with graph embeddings outperforms state-of-the-art models on long documents, showing significant improvements in F1 scores across all the datasets. Our study highlights the potential of GNN representations as a complementary approach to improve PLM performance for keyphrase extraction from long documents.

Related papers

• Scalable Weibull Graph Attention Autoencoder for Modeling Document Networks [50.42343781348247]
  We develop a graph Poisson factor analysis (GPFA) which provides analytic conditional posteriors to improve the inference accuracy. We also extend GPFA to a multi-stochastic-layer version named graph Poisson gamma belief network (GPGBN) to capture the hierarchical document relationships at multiple semantic levels. Our models can extract high-quality hierarchical latent document representations and achieve promising performance on various graph analytic tasks.
  arXiv  Detail & Related papers  (2024-10-13T02:22:14Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• Parameter-Efficient Tuning Large Language Models for Graph Representation Learning [62.26278815157628]
  We introduce Graph-aware. Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning. We use a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations.
  arXiv  Detail & Related papers  (2024-04-28T18:36:59Z)
• Empower Text-Attributed Graphs Learning with Large Language Models (LLMs) [5.920353954082262]
  We propose a plug-and-play approach to empower text-attributed graphs through node generation using Large Language Models (LLMs) We employ an edge predictor to capture the structural information inherent in the raw dataset and integrate the newly generated samples into the original graph. Experiments demonstrate the outstanding performance of our proposed paradigm, particularly in low-shot scenarios.
  arXiv  Detail & Related papers  (2023-10-15T16:04:28Z)
• SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning [131.04781590452308]
  We present SimTeG, a frustratingly Simple approach for Textual Graph learning. We first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task. We then generate node embeddings using the last hidden states of finetuned LM.
  arXiv  Detail & Related papers  (2023-08-03T07:00:04Z)
• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [51.90524745663737]
  A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks. Our method achieves state-of-the-art results on well-established TAG datasets. Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
  arXiv  Detail & Related papers  (2023-05-31T03:18:03Z)
• Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification Using Graph Neural Networks? [25.898812694174772]
  This work extensively investigates graph representation methods for text classification. We compare different graph construction schemes using a variety of GNN architectures and setups. Two Transformer-based large language models are also included to complement the study.
  arXiv  Detail & Related papers  (2023-05-23T23:31:24Z)
• Scientific Paper Extractive Summarization Enhanced by Citation Graphs [50.19266650000948]
  We focus on leveraging citation graphs to improve scientific paper extractive summarization under different settings. Preliminary results demonstrate that citation graph is helpful even in a simple unsupervised framework. Motivated by this, we propose a Graph-based Supervised Summarization model (GSS) to achieve more accurate results on the task when large-scale labeled data are available.
  arXiv  Detail & Related papers  (2022-12-08T11:53:12Z)
• Improving Graph-Based Text Representations with Character and Word Level N-grams [30.699644290131044]
  We propose a new word-character text graph that combines word and character n-gram nodes together with document nodes. We also propose two new graph-based neural models, WCTextGCN and WCTextGAT, for modeling our proposed text graph.
  arXiv  Detail & Related papers  (2022-10-12T08:07:54Z)
• ME-GCN: Multi-dimensional Edge-Embedded Graph Convolutional Networks for Semi-supervised Text Classification [6.196387205547024]
  This paper introduces the ME-GCN (Multi-dimensional Edge-enhanced Graph Convolutional Networks) for semi-supervised text classification. Our proposed model has significantly outperformed the state-of-the-art methods across eight benchmark datasets.
  arXiv  Detail & Related papers  (2022-04-10T07:05:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.