Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help
Multiple Graph Applications
- URL: http://arxiv.org/abs/2306.02592v1
- Date: Mon, 5 Jun 2023 04:46:44 GMT
- Title: Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help
Multiple Graph Applications
- Authors: Han Xie, Da Zheng, Jun Ma, Houyu Zhang, Vassilis N. Ioannidis, Xiang
Song, Qing Ping, Sheng Wang, Carl Yang, Yi Xu, Belinda Zeng, Trishul Chilimbi
- Abstract summary: We propose a framework of graph-aware language model pre-training on a large graph corpus.
We conduct experiments on Amazon's real internal datasets and large public datasets.
- Score: 38.83545631999851
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Model pre-training on large text corpora has been demonstrated effective for
various downstream applications in the NLP domain. In the graph mining domain,
a similar analogy can be drawn for pre-training graph models on large graphs in
the hope of benefiting downstream graph applications, which has also been
explored by several recent studies. However, no existing study has ever
investigated the pre-training of text plus graph models on large heterogeneous
graphs with abundant textual information (a.k.a. large graph corpora) and then
fine-tuning the model on different related downstream applications with
different graph schemas. To address this problem, we propose a framework of
graph-aware language model pre-training (GALM) on a large graph corpus, which
incorporates large language models and graph neural networks, and a variety of
fine-tuning methods on downstream applications. We conduct extensive
experiments on Amazon's real internal datasets and large public datasets.
Comprehensive empirical results and in-depth analysis demonstrate the
effectiveness of our proposed methods along with lessons learned.
Related papers
- Narrating Causal Graphs with Large Language Models [1.437446768735628]
This work explores the capability of large pretrained language models to generate text from causal graphs.
The causal reasoning encoded in these graphs can support applications as diverse as healthcare or marketing.
Results suggest users of generative AI can deploy future applications faster since similar performances are obtained when training a model with only a few examples.
arXiv Detail & Related papers (2024-03-11T19:19:59Z) - OpenGraph: Towards Open Graph Foundation Models [20.401374302429627]
We develop a general graph foundation model to understand the complex topological patterns present in diverse graph data.
We propose a unified graph tokenizer to adapt our graph model to generalize well on unseen graph data.
We also develop a scalable graph transformer, which effectively captures node-wise dependencies within the global topological context.
arXiv Detail & Related papers (2024-03-02T08:05:03Z) - Neural Scaling Laws on Graphs [54.435688297561015]
We study neural scaling laws on graphs from both model and data perspectives.
For model scaling, we investigate the phenomenon of scaling law collapse and identify overfitting as the potential reason.
For data scaling, we suggest that the number of graphs can not effectively metric the graph data volume in scaling law since the sizes of different graphs are highly irregular.
arXiv Detail & Related papers (2024-02-03T06:17:21Z) - Connecting the Dots: What Graph-Based Text Representations Work Best for
Text Classification Using Graph Neural Networks? [25.898812694174772]
This work extensively investigates graph representation methods for text classification.
We compare different graph construction schemes using a variety of GNN architectures and setups.
Two Transformer-based large language models are also included to complement the study.
arXiv Detail & Related papers (2023-05-23T23:31:24Z) - Connector 0.5: A unified framework for graph representation learning [5.398580049917152]
We introduce a novel graph representation framework covering various graph embedding models, ranging from shallow to state-of-the-art models.
We plan to build an efficient open-source framework that can provide deep graph embedding models to represent structural relations in graphs.
arXiv Detail & Related papers (2023-04-25T23:28:38Z) - Graph Learning and Its Advancements on Large Language Models: A Holistic
Survey [36.29355162928116]
This survey focuses on the most recent advancements in integrating graph learning with pre-trained language models.
We provide a holistic review that analyzes current works from the perspective of graph structure, and discusses the latest applications, trends, and challenges in graph learning.
arXiv Detail & Related papers (2022-12-17T22:05:07Z) - Similarity-aware Positive Instance Sampling for Graph Contrastive
Pre-training [82.68805025636165]
We propose to select positive graph instances directly from existing graphs in the training set.
Our selection is based on certain domain-specific pair-wise similarity measurements.
Besides, we develop an adaptive node-level pre-training method to dynamically mask nodes to distribute them evenly in the graph.
arXiv Detail & Related papers (2022-06-23T20:12:51Z) - Data Augmentation for Deep Graph Learning: A Survey [66.04015540536027]
We first propose a taxonomy for graph data augmentation and then provide a structured review by categorizing the related work based on the augmented information modalities.
Focusing on the two challenging problems in DGL (i.e., optimal graph learning and low-resource graph learning), we also discuss and review the existing learning paradigms which are based on graph data augmentation.
arXiv Detail & Related papers (2022-02-16T18:30:33Z) - GraphMI: Extracting Private Graph Data from Graph Neural Networks [59.05178231559796]
We present textbfGraph textbfModel textbfInversion attack (GraphMI), which aims to extract private graph data of the training graph by inverting GNN.
Specifically, we propose a projected gradient module to tackle the discreteness of graph edges while preserving the sparsity and smoothness of graph features.
We design a graph auto-encoder module to efficiently exploit graph topology, node attributes, and target model parameters for edge inference.
arXiv Detail & Related papers (2021-06-05T07:07:52Z) - Deep Learning for Learning Graph Representations [58.649784596090385]
Mining graph data has become a popular research topic in computer science.
The huge amount of network data has posed great challenges for efficient analysis.
This motivates the advent of graph representation which maps the graph into a low-dimension vector space.
arXiv Detail & Related papers (2020-01-02T02:13:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.