JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs

• URL: http://arxiv.org/abs/2106.10502v1
• Date: Sat, 19 Jun 2021 14:10:10 GMT
• Title: JointGT: Graph-Text Joint Representation Learning for Text Generation from Knowledge Graphs
• Authors: Pei Ke, Haozhe Ji, Yu Ran, Xin Cui, Liwei Wang, Linfeng Song, Xiaoyan Zhu, Minlie Huang
• Abstract summary: We propose a graph-text joint representation learning model called JointGT. During encoding, we devise a structure-aware semantic aggregation module which is plugged into each Transformer layer. We show that JointGT obtains new state-of-the-art performance on various KG-to-text datasets.
• Score: 44.06715423776722
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Existing pre-trained models for knowledge-graph-to-text (KG-to-text) generation simply fine-tune text-to-text pre-trained models such as BART or T5 on KG-to-text datasets, which largely ignore the graph structure during encoding and lack elaborate pre-training tasks to explicitly model graph-text alignments. To tackle these problems, we propose a graph-text joint representation learning model called JointGT. During encoding, we devise a structure-aware semantic aggregation module which is plugged into each Transformer layer to preserve the graph structure. Furthermore, we propose three new pre-training tasks to explicitly enhance the graph-text alignment including respective text / graph reconstruction, and graph-text alignment in the embedding space via Optimal Transport. Experiments show that JointGT obtains new state-of-the-art performance on various KG-to-text datasets.

Related papers

• Hierarchical Compression of Text-Rich Graphs via Large Language Models [63.75293588479027]
  Text-rich graphs are prevalent in data mining contexts like e-commerce and academic graphs. This paper introduces Hierarchical Compression'' (HiCom), a novel method to align the capabilities of LLMs with the structure of text-rich graphs. HiCom can outperform both GNNs and LLM backbones for node classification on e-commerce and citation graphs.
  arXiv  Detail & Related papers  (2024-06-13T07:24:46Z)
• TAGA: Text-Attributed Graph Self-Supervised Learning by Synergizing Graph and Text Mutual Transformations [15.873944819608434]
  Text-Attributed Graphs (TAGs) enhance graph structures with natural language descriptions. This paper introduces a new self-supervised learning framework, Text-And-Graph Multi-View Alignment (TAGA), which integrates TAGs' structural and semantic dimensions. Our framework demonstrates strong performance in zero-shot and few-shot scenarios across eight real-world datasets.
  arXiv  Detail & Related papers  (2024-05-27T03:40:16Z)
• Pretraining Language Models with Text-Attributed Heterogeneous Graphs [28.579509154284448]
  We present a new pretraining framework for Language Models (LMs) that explicitly considers the topological and heterogeneous information in Text-Attributed Heterogeneous Graphs (TAHGs) We propose a topology-aware pretraining task to predict nodes involved in the context graph by jointly optimizing an LM and an auxiliary heterogeneous graph neural network. We conduct link prediction and node classification tasks on three datasets from various domains.
  arXiv  Detail & Related papers  (2023-10-19T08:41:21Z)
• Improving Graph-Based Text Representations with Character and Word Level N-grams [30.699644290131044]
  We propose a new word-character text graph that combines word and character n-gram nodes together with document nodes. We also propose two new graph-based neural models, WCTextGCN and WCTextGAT, for modeling our proposed text graph.
  arXiv  Detail & Related papers  (2022-10-12T08:07:54Z)
• Stage-wise Fine-tuning for Graph-to-Text Generation [25.379346921398326]
  Graph-to-text generation has benefited from pre-trained language models (PLMs) in achieving better performance than structured graph encoders. We propose a structured graph-to-text model with a two-step fine-tuning mechanism which first fine-tunes model on Wikipedia before adapting to the graph-to-text generation.
  arXiv  Detail & Related papers  (2021-05-17T17:15:29Z)
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph [53.70520466556453]
  We propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow. In addition, a progressive learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph.
  arXiv  Detail & Related papers  (2021-05-06T12:20:41Z)
• Structural Adapters in Pretrained Language Models for AMR-to-text Generation [59.50420985074769]
  Previous work on text generation from graph-structured data relies on pretrained language models (PLMs) We propose StructAdapt, an adapter method to encode graph structure into PLMs.
  arXiv  Detail & Related papers  (2021-03-16T15:06:50Z)
• Promoting Graph Awareness in Linearized Graph-to-Text Generation [72.83863719868364]
  We study the ability of linearized models to encode local graph structures. Our findings motivate solutions to enrich the quality of models' implicit graph encodings. We find that these denoising scaffolds lead to substantial improvements in downstream generation in low-resource settings.
  arXiv  Detail & Related papers  (2020-12-31T18:17:57Z)
• Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoning [72.52804406378023]
  Cross-modal retrieval between videos and texts has attracted growing attentions due to the rapid emergence of videos on the web. To improve fine-grained video-text retrieval, we propose a Hierarchical Graph Reasoning model, which decomposes video-text matching into global-to-local levels.
  arXiv  Detail & Related papers  (2020-03-01T03:44:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.