Revisiting Inferential Benchmarks for Knowledge Graph Completion

• URL: http://arxiv.org/abs/2306.04814v1
• Date: Wed, 7 Jun 2023 22:35:39 GMT
• Title: Revisiting Inferential Benchmarks for Knowledge Graph Completion
• Authors: Shuwen Liu, Bernardo Cuenca Grau, Ian Horrocks, Egor V. Kostylev
• Abstract summary: Key feature of Machine Learning approaches for Knowledge Graph (KG) completion is their ability to learn inference patterns. Standard completion benchmarks are not well-suited for evaluating models' abilities to learn patterns. We propose a novel approach for designing KG completion benchmarks based on the following principles.
• Score: 29.39724559354927
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge Graph (KG) completion is the problem of extending an incomplete KG with missing facts. A key feature of Machine Learning approaches for KG completion is their ability to learn inference patterns, so that the predicted facts are the results of applying these patterns to the KG. Standard completion benchmarks, however, are not well-suited for evaluating models' abilities to learn patterns, because the training and test sets of these benchmarks are a random split of a given KG and hence do not capture the causality of inference patterns. We propose a novel approach for designing KG completion benchmarks based on the following principles: there is a set of logical rules so that the missing facts are the results of the rules' application; the training set includes both premises matching rule antecedents and the corresponding conclusions; the test set consists of the results of applying the rules to the training set; the negative examples are designed to discourage the models from learning rules not entailed by the rule set. We use our methodology to generate several benchmarks and evaluate a wide range of existing KG completion systems. Our results provide novel insights on the ability of existing models to induce inference patterns from incomplete KGs.

Related papers

• Learning Rules from KGs Guided by Language Models [48.858741745144044]
  Rule learning methods can be applied to predict potentially missing facts. Ranking of rules is especially challenging over highly incomplete or biased KGs. With the recent rise of Language Models (LMs) several works have claimed that LMs can be used as alternative means for KG completion.
  arXiv  Detail & Related papers  (2024-09-12T09:27:36Z)
• Retrieved In-Context Principles from Previous Mistakes [55.109234526031884]
  In-context learning (ICL) has been instrumental in adapting Large Language Models (LLMs) to downstream tasks using correct input-output examples. Recent advances have attempted to improve model performance through principles derived from mistakes. We propose Retrieved In-Context Principles (RICP), a novel teacher-student framework.
  arXiv  Detail & Related papers  (2024-07-08T07:32:26Z)
• On Training Survival Models with Scoring Rules [9.330089124239086]
  This work investigates using scoring rules for model training rather than evaluation. We establish a general framework for training survival models that is model agnostic and can learn event time distributions parametrically or non-parametrically. Empirical comparisons on synthetic and real-world data indicate that scoring rules can be successfully incorporated into model training.
  arXiv  Detail & Related papers  (2024-03-19T20:58:38Z)
• Understanding prompt engineering may not require rethinking generalization [56.38207873589642]
  We show that the discrete nature of prompts, combined with a PAC-Bayes prior given by a language model, results in generalization bounds that are remarkably tight by the standards of the literature. This work provides a possible justification for the widespread practice of prompt engineering.
  arXiv  Detail & Related papers  (2023-10-06T00:52:48Z)
• KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models [76.01814380927507]
  KGxBoard is an interactive framework for performing fine-grained evaluation on meaningful subsets of the data. In our experiments, we highlight the findings with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.
  arXiv  Detail & Related papers  (2022-08-23T15:11:45Z)
• Improving Knowledge Graph Representation Learning by Structure Contextual Pre-training [9.70121995251553]
  We propose a novel pre-training-then-fine-tuning framework for knowledge graph representation learning. A KG model is pre-trained with triple classification task, followed by discriminative fine-tuning on specific downstream tasks. Experimental results demonstrate that fine-tuning SCoP not only outperforms results of baselines on a portfolio of downstream tasks but also avoids tedious task-specific model design and parameter training.
  arXiv  Detail & Related papers  (2021-12-08T02:50:54Z)
• EngineKGI: Closed-Loop Knowledge Graph Inference [37.15381932994768]
  EngineKGI is a novel closed-loop KG inference framework. It combines KGE and rule learning to complement each other in a closed-loop pattern. Our model outperforms other baselines on link prediction tasks.
  arXiv  Detail & Related papers  (2021-12-02T08:02:59Z)
• The MultiBERTs: BERT Reproductions for Robustness Analysis [86.29162676103385]
  Re-running pretraining can lead to substantially different conclusions about performance. We introduce MultiBERTs: a set of 25 BERT-base checkpoints. The aim is to enable researchers to draw robust and statistically justified conclusions about pretraining procedures.
  arXiv  Detail & Related papers  (2021-06-30T15:56:44Z)
• Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning [61.32992639292889]
  Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks. We introduce a new scoring method that casts a plausibility ranking task in a full-text format. We show that our method provides a much more stable training phase across random restarts.
  arXiv  Detail & Related papers  (2020-04-29T10:54:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.