Towards Foundation Models for Knowledge Graph Reasoning

• URL: http://arxiv.org/abs/2310.04562v2
• Date: Tue, 9 Apr 2024 19:48:22 GMT
• Title: Towards Foundation Models for Knowledge Graph Reasoning
• Authors: Mikhail Galkin, Xinyu Yuan, Hesham Mostafa, Jian Tang, Zhaocheng Zhu,
• Abstract summary: Knowledge graphs (KGs) have different entity and relation vocabularies that generally do not overlap. We present ULTRA, an approach for learning universal and transferable graph representations. We find that the zero-shot inductive inference performance of a single pre-trained ULTRA model on unseen graphs of various sizes is often on par or better than strong baselines trained on specific graphs.
• Score: 18.77355708537997
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Foundation models in language and vision have the ability to run inference on any textual and visual inputs thanks to the transferable representations such as a vocabulary of tokens in language. Knowledge graphs (KGs) have different entity and relation vocabularies that generally do not overlap. The key challenge of designing foundation models on KGs is to learn such transferable representations that enable inference on any graph with arbitrary entity and relation vocabularies. In this work, we make a step towards such foundation models and present ULTRA, an approach for learning universal and transferable graph representations. ULTRA builds relational representations as a function conditioned on their interactions. Such a conditioning strategy allows a pre-trained ULTRA model to inductively generalize to any unseen KG with any relation vocabulary and to be fine-tuned on any graph. Conducting link prediction experiments on 57 different KGs, we find that the zero-shot inductive inference performance of a single pre-trained ULTRA model on unseen graphs of various sizes is often on par or better than strong baselines trained on specific graphs. Fine-tuning further boosts the performance.

Related papers

• Towards Graph Foundation Models: The Perspective of Zero-shot Reasoning on Knowledge Graphs [14.392577069212292]
  We introduce SCORE, a unified graph reasoning framework that effectively generalizes diverse graph tasks using zero-shot learning. We evaluate SCORE using 38 diverse graph datasets, covering node-level, link-level, and graph-level tasks across multiple domains.
  arXiv  Detail & Related papers  (2024-10-16T14:26:08Z)
• Verbalized Graph Representation Learning: A Fully Interpretable Graph Model Based on Large Language Models Throughout the Entire Process [8.820909397907274]
  We propose a verbalized graph representation learning (VGRL) method which is fully interpretable. In contrast to traditional graph machine learning models, VGRL constrains this parameter space to be text description. We conduct several studies to empirically evaluate the effectiveness of VGRL.
  arXiv  Detail & Related papers  (2024-10-02T12:07:47Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• UniGraph: Learning a Unified Cross-Domain Foundation Model for Text-Attributed Graphs [30.635472655668078]
  Text-Attributed Graphs (TAGs) can generalize to unseen graphs and tasks across diverse domains. We propose a novel cascaded architecture of Language Models (LMs) and Graph Neural Networks (GNNs) as backbone networks. We demonstrate the model's effectiveness in self-supervised representation learning on unseen graphs, few-shot in-context transfer, and zero-shot transfer.
  arXiv  Detail & Related papers  (2024-02-21T09:06:31Z)
• When Graph Data Meets Multimodal: A New Paradigm for Graph Understanding and Reasoning [54.84870836443311]
  The paper presents a new paradigm for understanding and reasoning about graph data by integrating image encoding and multimodal technologies. This approach enables the comprehension of graph data through an instruction-response format, utilizing GPT-4V's advanced capabilities. The study evaluates this paradigm on various graph types, highlighting the model's strengths and weaknesses, particularly in Chinese OCR performance and complex reasoning tasks.
  arXiv  Detail & Related papers  (2023-12-16T08:14:11Z)
• Hi-ArG: Exploring the Integration of Hierarchical Argumentation Graphs in Language Pretraining [62.069374456021016]
  We propose the Hierarchical Argumentation Graph (Hi-ArG), a new structure to organize arguments. We also introduce two approaches to exploit Hi-ArG, including a text-graph multi-modal model GreaseArG and a new pre-training framework augmented with graph information.
  arXiv  Detail & Related papers  (2023-12-01T19:03:38Z)
• One for All: Towards Training One Graph Model for All Classification Tasks [61.656962278497225]
  A unified model for various graph tasks remains underexplored, primarily due to the challenges unique to the graph learning domain. We propose textbfOne for All (OFA), the first general framework that can use a single graph model to address the above challenges. OFA performs well across different tasks, making it the first general-purpose across-domains classification model on graphs.
  arXiv  Detail & Related papers  (2023-09-29T21:15:26Z)
• ConGraT: Self-Supervised Contrastive Pretraining for Joint Graph and Text Embeddings [20.25180279903009]
  We propose Contrastive Graph-Text pretraining (ConGraT) for jointly learning separate representations of texts and nodes in a text-attributed graph (TAG) Our method trains a language model (LM) and a graph neural network (GNN) to align their representations in a common latent space using a batch-wise contrastive learning objective inspired by CLIP. Experiments demonstrate that ConGraT outperforms baselines on various downstream tasks, including node and text category classification, link prediction, and language modeling.
  arXiv  Detail & Related papers  (2023-05-23T17:53:30Z)
• I Know What You Do Not Know: Knowledge Graph Embedding via Co-distillation Learning [16.723470319188102]
  Knowledge graph embedding seeks to learn vector representations for entities and relations. Recent studies have used pre-trained language models to learn embeddings based on the textual information of entities and relations. We propose CoLE, a Co-distillation Learning method for KG Embedding that exploits the complement of graph structures and text information.
  arXiv  Detail & Related papers  (2022-08-21T07:34:37Z)
• Explanation Graph Generation via Pre-trained Language Models: An Empirical Study with Contrastive Learning [84.35102534158621]
  We study pre-trained language models that generate explanation graphs in an end-to-end manner. We propose simple yet effective ways of graph perturbations via node and edge edit operations. Our methods lead to significant improvements in both structural and semantic accuracy of explanation graphs.
  arXiv  Detail & Related papers  (2022-04-11T00:58:27Z)
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph [53.70520466556453]
  We propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow. In addition, a progressive learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph.
  arXiv  Detail & Related papers  (2021-05-06T12:20:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.