Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?

• URL: http://arxiv.org/abs/2311.09109v2
• Date: Thu, 6 Jun 2024 16:01:08 GMT
• Title: Does Pre-trained Language Model Actually Infer Unseen Links in Knowledge Graph Completion?
• Authors: Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe,
• Abstract summary: Knowledge graphs (KGs) consist of links that describe relationships between entities. Knowledge Graph Completion (KGC) is a task that infers unseen relationships between entities in a KG. Traditional embedding-based KGC methods, such as RESCAL, infer missing links using only the knowledge from training data. Recent Pre-trained Language Model (PLM)-based KGC utilizes knowledge obtained during pre-training.
• Score: 32.645448509968226
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge graphs (KGs) consist of links that describe relationships between entities. Due to the difficulty of manually enumerating all relationships between entities, automatically completing them is essential for KGs. Knowledge Graph Completion (KGC) is a task that infers unseen relationships between entities in a KG. Traditional embedding-based KGC methods, such as RESCAL, TransE, DistMult, ComplEx, RotatE, HAKE, HousE, etc., infer missing links using only the knowledge from training data. In contrast, the recent Pre-trained Language Model (PLM)-based KGC utilizes knowledge obtained during pre-training. Therefore, PLM-based KGC can estimate missing links between entities by reusing memorized knowledge from pre-training without inference. This approach is problematic because building KGC models aims to infer unseen links between entities. However, conventional evaluations in KGC do not consider inference and memorization abilities separately. Thus, a PLM-based KGC method, which achieves high performance in current KGC evaluations, may be ineffective in practical applications. To address this issue, we analyze whether PLM-based KGC methods make inferences or merely access memorized knowledge. For this purpose, we propose a method for constructing synthetic datasets specified in this analysis and conclude that PLMs acquire the inference abilities required for KGC through pre-training, even though the performance improvements mostly come from textual information of entities and relations.

Related papers

• A Knowledge-Injected Curriculum Pretraining Framework for Question Answering [70.13026036388794]
  We propose a general Knowledge-Injected Curriculum Pretraining framework (KICP) to achieve comprehensive KG learning and exploitation for Knowledge-based question answering tasks. The KI module first injects knowledge into the LM by generating KG-centered pretraining corpus, and generalizes the process into three key steps. The KA module learns knowledge from the generated corpus with LM equipped with an adapter as well as keeps its original natural language understanding ability. The CR module follows human reasoning patterns to construct three corpora with increasing difficulties of reasoning, and further trains the LM from easy to hard in a curriculum manner.
  arXiv  Detail & Related papers  (2024-03-11T03:42:03Z)
• Multi-perspective Improvement of Knowledge Graph Completion with Large Language Models [95.31941227776711]
  We propose MPIKGC to compensate for the deficiency of contextualized knowledge and improve KGC by querying large language models (LLMs) We conducted extensive evaluation of our framework based on four description-based KGC models and four datasets, for both link prediction and triplet classification tasks.
  arXiv  Detail & Related papers  (2024-03-04T12:16:15Z)
• KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion [27.405080941584533]
  We propose KICGPT, a framework that integrates a large language model and a triple-based KGC retriever. It alleviates the long-tail problem without incurring additional training overhead. Empirical results on benchmark datasets demonstrate the effectiveness of KICGPT with smaller training overhead and no finetuning.
  arXiv  Detail & Related papers  (2024-02-04T08:01:07Z)
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion [51.126166442122546]
  We introduce the Contextualization Distillation strategy, a plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models to transform compact, structural triplets into context-rich segments. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach.
  arXiv  Detail & Related papers  (2024-01-28T08:56:49Z)
• Prompting Disentangled Embeddings for Knowledge Graph Completion with Pre-trained Language Model [38.00241874974804]
  Both graph structures and textual information play a critical role in Knowledge Graph Completion (KGC) We propose a new KGC method named PDKGC with two prompts -- a hard task prompt and a disentangled structure prompt. With the two prompts, PDKGC builds a textual predictor and a structural predictor, respectively, and their combination leads to more comprehensive entity prediction.
  arXiv  Detail & Related papers  (2023-12-04T12:20:25Z)
• Enhancing Text-based Knowledge Graph Completion with Zero-Shot Large Language Models: A Focus on Semantic Enhancement [8.472388165833292]
  We introduce a framework termed constrained prompts for KGC (CP-KGC) This framework designs prompts that adapt to different datasets to enhance semantic richness. This study extends the performance limits of existing models and promotes further integration of KGC with large language models.
  arXiv  Detail & Related papers  (2023-10-12T12:31:23Z)
• Improving Few-Shot Inductive Learning on Temporal Knowledge Graphs using Confidence-Augmented Reinforcement Learning [24.338098716004485]
  TKGC aims to predict the missing links among the entities in a temporal knwoledge graph (TKG) Recently, a new task, i.e., TKG few-shot out-of-graph (OOG) link prediction, is proposed. We propose a TKGC method FITCARL that combines few-shot learning with reinforcement learning to solve this task.
  arXiv  Detail & Related papers  (2023-04-02T20:05:20Z)
• BertNet: Harvesting Knowledge Graphs with Arbitrary Relations from Pretrained Language Models [65.51390418485207]
  We propose a new approach of harvesting massive KGs of arbitrary relations from pretrained LMs. With minimal input of a relation definition, the approach efficiently searches in the vast entity pair space to extract diverse accurate knowledge. We deploy the approach to harvest KGs of over 400 new relations from different LMs.
  arXiv  Detail & Related papers  (2022-06-28T19:46:29Z)
• Supporting Vision-Language Model Inference with Confounder-pruning Knowledge Prompt [71.77504700496004]
  Vision-language models are pre-trained by aligning image-text pairs in a common space to deal with open-set visual concepts. To boost the transferability of the pre-trained models, recent works adopt fixed or learnable prompts. However, how and what prompts can improve inference performance remains unclear.
  arXiv  Detail & Related papers  (2022-05-23T07:51:15Z)
• EngineKGI: Closed-Loop Knowledge Graph Inference [37.15381932994768]
  EngineKGI is a novel closed-loop KG inference framework. It combines KGE and rule learning to complement each other in a closed-loop pattern. Our model outperforms other baselines on link prediction tasks.
  arXiv  Detail & Related papers  (2021-12-02T08:02:59Z)
• Generative Adversarial Zero-Shot Relational Learning for Knowledge Graphs [96.73259297063619]
  We consider a novel formulation, zero-shot learning, to free this cumbersome curation. For newly-added relations, we attempt to learn their semantic features from their text descriptions. We leverage Generative Adrial Networks (GANs) to establish the connection between text and knowledge graph domain.
  arXiv  Detail & Related papers  (2020-01-08T01:19:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.