Graph Pre-training for AMR Parsing and Generation

• URL: http://arxiv.org/abs/2203.07836v1
• Date: Tue, 15 Mar 2022 12:47:00 GMT
• Title: Graph Pre-training for AMR Parsing and Generation
• Authors: Xuefeng Bai, Yulong Chen, Yue Zhang
• Abstract summary: We investigate graph self-supervised training to improve structure awareness of PLMs over AMR graphs. We introduce two graph auto-encoding strategies for graph-to-graph pre-training and four tasks to integrate text and graph information during pre-training.
• Score: 14.228434699363495
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Abstract meaning representation (AMR) highlights the core semantic information of text in a graph structure. Recently, pre-trained language models (PLMs) have advanced tasks of AMR parsing and AMR-to-text generation, respectively. However, PLMs are typically pre-trained on textual data, thus are sub-optimal for modeling structural knowledge. To this end, we investigate graph self-supervised training to improve the structure awareness of PLMs over AMR graphs. In particular, we introduce two graph auto-encoding strategies for graph-to-graph pre-training and four tasks to integrate text and graph information during pre-training. We further design a unified framework to bridge the gap between pre-training and fine-tuning tasks. Experiments on both AMR parsing and AMR-to-text generation show the superiority of our model. To our knowledge, we are the first to consider pre-training on semantic graphs.

Related papers

• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning [131.04781590452308]
  We present SimTeG, a frustratingly Simple approach for Textual Graph learning. We first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task. We then generate node embeddings using the last hidden states of finetuned LM.
  arXiv  Detail & Related papers  (2023-08-03T07:00:04Z)
• Incorporating Graph Information in Transformer-based AMR Parsing [34.461828101932184]
  LeakDistill is a model and method that explores a modification to the Transformer architecture. We show how, by employing word-to-node alignment to embed graph structural information into the encoder at training time, we can obtain state-of-the-art AMR parsing.
  arXiv  Detail & Related papers  (2023-06-23T12:12:08Z)
• PRODIGY: Enabling In-context Learning Over Graphs [112.19056551153454]
  In-context learning is the ability of a pretrained model to adapt to novel and diverse downstream tasks. We develop PRODIGY, the first pretraining framework that enables in-context learning over graphs.
  arXiv  Detail & Related papers  (2023-05-21T23:16:30Z)
• Self-supervised Graph Masking Pre-training for Graph-to-Text Generation [5.108327983929205]
  Large-scale pre-trained language models (PLMs) have advanced Graph-to-Text (G2T) generation. We propose graph masking pre-training strategies that neither require supervision signals nor adjust the architecture of the underlying pre-trained encoder-decoder model. Our approach achieves new state-of-the-art results on WebNLG+ 2020 and EventNarrative G2T generation datasets.
  arXiv  Detail & Related papers  (2022-10-19T14:44:56Z)
• GAP: A Graph-aware Language Model Framework for Knowledge Graph-to-Text Generation [3.593955557310285]
  Recent improvements in KG-to-text generation are due to auxiliary pre-training tasks designed to give the fine-tune task a boost in performance. Here, we demonstrate that by fusing graph-aware elements into existing pre-trained language models, we are able to outperform state-of-the-art models and close the gap imposed by additional pre-training tasks.
  arXiv  Detail & Related papers  (2022-04-13T23:53:37Z)
• Neural Graph Matching for Pre-training Graph Neural Networks [72.32801428070749]
  Graph neural networks (GNNs) have been shown powerful capacity at modeling structural data. We present a novel Graph Matching based GNN Pre-Training framework, called GMPT. The proposed method can be applied to fully self-supervised pre-training and coarse-grained supervised pre-training.
  arXiv  Detail & Related papers  (2022-03-03T09:53:53Z)
• Structural Adapters in Pretrained Language Models for AMR-to-text Generation [59.50420985074769]
  Previous work on text generation from graph-structured data relies on pretrained language models (PLMs) We propose StructAdapt, an adapter method to encode graph structure into PLMs.
  arXiv  Detail & Related papers  (2021-03-16T15:06:50Z)
• Investigating Pretrained Language Models for Graph-to-Text Generation [55.55151069694146]
  Graph-to-text generation aims to generate fluent texts from graph-based data. We present a study across three graph domains: meaning representations, Wikipedia knowledge graphs (KGs) and scientific KGs. We show that the PLMs BART and T5 achieve new state-of-the-art results and that task-adaptive pretraining strategies improve their performance even further.
  arXiv  Detail & Related papers  (2020-07-16T16:05:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.