Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications

• URL: http://arxiv.org/abs/2306.02592v1
• Date: Mon, 5 Jun 2023 04:46:44 GMT
• Title: Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications
• Authors: Han Xie, Da Zheng, Jun Ma, Houyu Zhang, Vassilis N. Ioannidis, Xiang Song, Qing Ping, Sheng Wang, Carl Yang, Yi Xu, Belinda Zeng, Trishul Chilimbi
• Abstract summary: We propose a framework of graph-aware language model pre-training on a large graph corpus. We conduct experiments on Amazon's real internal datasets and large public datasets.
• Score: 38.83545631999851
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Model pre-training on large text corpora has been demonstrated effective for various downstream applications in the NLP domain. In the graph mining domain, a similar analogy can be drawn for pre-training graph models on large graphs in the hope of benefiting downstream graph applications, which has also been explored by several recent studies. However, no existing study has ever investigated the pre-training of text plus graph models on large heterogeneous graphs with abundant textual information (a.k.a. large graph corpora) and then fine-tuning the model on different related downstream applications with different graph schemas. To address this problem, we propose a framework of graph-aware language model pre-training (GALM) on a large graph corpus, which incorporates large language models and graph neural networks, and a variety of fine-tuning methods on downstream applications. We conduct extensive experiments on Amazon's real internal datasets and large public datasets. Comprehensive empirical results and in-depth analysis demonstrate the effectiveness of our proposed methods along with lessons learned.

Related papers

• Parametric Graph Representations in the Era of Foundation Models: A Survey and Position [69.48708136448694]
  Graphs have been widely used in the past decades of big data and AI to model comprehensive relational data. Identifying meaningful graph laws can significantly enhance the effectiveness of various applications.
  arXiv  Detail & Related papers  (2024-10-16T00:01:31Z)
• Does Graph Prompt Work? A Data Operation Perspective with Theoretical Analysis [7.309233340654514]
  This paper introduces a theoretical framework that rigorously analyzes graph prompting from a data operation perspective. We provide a formal guarantee theorem, demonstrating graph prompts capacity to approximate graph transformation operators. We derive upper bounds on the error of these data operations by graph prompts for a single graph and extend this discussion to batches of graphs.
  arXiv  Detail & Related papers  (2024-10-02T15:07:13Z)
• Narrating Causal Graphs with Large Language Models [1.437446768735628]
  This work explores the capability of large pretrained language models to generate text from causal graphs. The causal reasoning encoded in these graphs can support applications as diverse as healthcare or marketing. Results suggest users of generative AI can deploy future applications faster since similar performances are obtained when training a model with only a few examples.
  arXiv  Detail & Related papers  (2024-03-11T19:19:59Z)
• Neural Scaling Laws on Graphs [54.435688297561015]
  We study neural scaling laws on graphs from both model and data perspectives. For model scaling, we investigate the phenomenon of scaling law collapse and identify overfitting as the potential reason. For data scaling, we suggest that the number of graphs can not effectively metric the graph data volume in scaling law since the sizes of different graphs are highly irregular.
  arXiv  Detail & Related papers  (2024-02-03T06:17:21Z)
• Connecting the Dots: What Graph-Based Text Representations Work Best for Text Classification Using Graph Neural Networks? [25.898812694174772]
  This work extensively investigates graph representation methods for text classification. We compare different graph construction schemes using a variety of GNN architectures and setups. Two Transformer-based large language models are also included to complement the study.
  arXiv  Detail & Related papers  (2023-05-23T23:31:24Z)
• Graph Learning and Its Advancements on Large Language Models: A Holistic Survey [37.01696685233113]
  This survey focuses on the most recent advancements in integrating graph learning with pre-trained language models. We provide a holistic review that analyzes current works from the perspective of graph structure, and discusses the latest applications, trends, and challenges in graph learning.
  arXiv  Detail & Related papers  (2022-12-17T22:05:07Z)
• Similarity-aware Positive Instance Sampling for Graph Contrastive Pre-training [82.68805025636165]
  We propose to select positive graph instances directly from existing graphs in the training set. Our selection is based on certain domain-specific pair-wise similarity measurements. Besides, we develop an adaptive node-level pre-training method to dynamically mask nodes to distribute them evenly in the graph.
  arXiv  Detail & Related papers  (2022-06-23T20:12:51Z)
• Data Augmentation for Deep Graph Learning: A Survey [66.04015540536027]
  We first propose a taxonomy for graph data augmentation and then provide a structured review by categorizing the related work based on the augmented information modalities. Focusing on the two challenging problems in DGL (i.e., optimal graph learning and low-resource graph learning), we also discuss and review the existing learning paradigms which are based on graph data augmentation.
  arXiv  Detail & Related papers  (2022-02-16T18:30:33Z)
• GraphMI: Extracting Private Graph Data from Graph Neural Networks [59.05178231559796]
  We present textbfGraph textbfModel textbfInversion attack (GraphMI), which aims to extract private graph data of the training graph by inverting GNN. Specifically, we propose a projected gradient module to tackle the discreteness of graph edges while preserving the sparsity and smoothness of graph features. We design a graph auto-encoder module to efficiently exploit graph topology, node attributes, and target model parameters for edge inference.
  arXiv  Detail & Related papers  (2021-06-05T07:07:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.