KnowFormer: Revisiting Transformers for Knowledge Graph Reasoning

• URL: http://arxiv.org/abs/2409.12865v1
• Date: Thu, 19 Sep 2024 16:08:10 GMT
• Title: KnowFormer: Revisiting Transformers for Knowledge Graph Reasoning
• Authors: Junnan Liu, Qianren Mao, Weifeng Jiang, Jianxin Li,
• Abstract summary: We propose KnowFormer.KnowFormer to perform reasoning on knowledge graphs from the message-passing perspective. To incorporate structural information into the self-attention mechanism, we introduce structure-aware modules to calculate query, key, and value. Experimental results demonstrate the superior performance of KnowFormer compared to prominent baseline methods on both transductive and inductive benchmarks.
• Score: 10.445709698341682
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge graph reasoning plays a vital role in various applications and has garnered considerable attention. Recently, path-based methods have achieved impressive performance. However, they may face limitations stemming from constraints in message-passing neural networks, such as missing paths and information over-squashing. In this paper, we revisit the application of transformers for knowledge graph reasoning to address the constraints faced by path-based methods and propose a novel method KnowFormer.KnowFormer utilizes a transformer architecture to perform reasoning on knowledge graphs from the message-passing perspective, rather than reasoning by textual information like previous pretrained language model based methods. Specifically, we define the attention computation based on the query prototype of knowledge graph reasoning, facilitating convenient construction and efficient optimization. To incorporate structural information into the self-attention mechanism, we introduce structure-aware modules to calculate query, key, and value respectively. Additionally, we present an efficient attention computation method for better scalability. Experimental results demonstrate the superior performance of KnowFormer compared to prominent baseline methods on both transductive and inductive benchmarks.

Related papers

• Question-guided Knowledge Graph Re-scoring and Injection for Knowledge Graph Question Answering [27.414670144354453]
  KGQA involves answering natural language questions by leveraging structured information stored in a knowledge graph. We propose a Question-guided Knowledge Graph Re-scoring method (Q-KGR) to eliminate noisy pathways for the input question. We also introduce Knowformer, a parameter-efficient method for injecting the re-scored knowledge graph into large language models to enhance their ability to perform factual reasoning.
  arXiv  Detail & Related papers  (2024-10-02T10:27:07Z)
• Extending Transductive Knowledge Graph Embedding Models for Inductive Logical Relational Inference [0.5439020425819]
  This work bridges the gap between traditional transductive knowledge graph embedding approaches and more recent inductive relation prediction models. We introduce a generalized form of harmonic extension which leverages representations learned through transductive embedding methods to infer representations of new entities introduced at inference time as in the inductive setting. In experiments on a number of large-scale knowledge graph embedding benchmarks, we find that this approach for extending the functionality of transductive knowledge graph embedding models is competitive with--and in some scenarios outperforms--several state-of-the-art models derived explicitly for such inductive tasks.
  arXiv  Detail & Related papers  (2023-09-07T15:24:18Z)
• Schema-aware Reference as Prompt Improves Data-Efficient Knowledge Graph Construction [57.854498238624366]
  We propose a retrieval-augmented approach, which retrieves schema-aware Reference As Prompt (RAP) for data-efficient knowledge graph construction. RAP can dynamically leverage schema and knowledge inherited from human-annotated and weak-supervised data as a prompt for each sample.
  arXiv  Detail & Related papers  (2022-10-19T16:40:28Z)
• Joint Language Semantic and Structure Embedding for Knowledge Graph Completion [66.15933600765835]
  We propose to jointly embed the semantics in the natural language description of the knowledge triplets with their structure information. Our method embeds knowledge graphs for the completion task via fine-tuning pre-trained language models. Our experiments on a variety of knowledge graph benchmarks have demonstrated the state-of-the-art performance of our method.
  arXiv  Detail & Related papers  (2022-09-19T02:41:02Z)
• Learning Graph Regularisation for Guided Super-Resolution [77.7568596501908]
  We introduce a novel formulation for guided super-resolution. Its core is a differentiable optimisation layer that operates on a learned affinity graph. We extensively evaluate our method on several datasets, and consistently outperform recent baselines in terms of quantitative reconstruction errors.
  arXiv  Detail & Related papers  (2022-03-27T13:12:18Z)
• MERIt: Meta-Path Guided Contrastive Learning for Logical Reasoning [63.50909998372667]
  We propose MERIt, a MEta-path guided contrastive learning method for logical ReasonIng of text. Two novel strategies serve as indispensable components of our method.
  arXiv  Detail & Related papers  (2022-03-01T11:13:00Z)
• Information fusion between knowledge and data in Bayesian network structure learning [5.994412766684843]
  This paper describes and evaluates a set of information fusion methods that have been implemented in the open-source Bayesys structure learning system. The results are illustrated both with limited and big data, with application to three BN structure learning algorithms available in Bayesys.
  arXiv  Detail & Related papers  (2021-01-31T15:45:29Z)
• Information Theoretic Meta Learning with Gaussian Processes [74.54485310507336]
  We formulate meta learning using information theoretic concepts; namely, mutual information and the information bottleneck. By making use of variational approximations to the mutual information, we derive a general and tractable framework for meta learning.
  arXiv  Detail & Related papers  (2020-09-07T16:47:30Z)
• Benchmarking neural embeddings for link prediction in knowledge graphs under semantic and structural changes [6.23228063561537]
  We propose an open-source evaluation pipeline, which benchmarks the accuracy of neural embeddings. We define relation-centric connectivity measures that allow us to connect the link prediction capacity to the structure of the knowledge graph.
  arXiv  Detail & Related papers  (2020-05-15T17:15:45Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.