Revisiting Inferential Benchmarks for Knowledge Graph Completion
- URL: http://arxiv.org/abs/2306.04814v1
- Date: Wed, 7 Jun 2023 22:35:39 GMT
- Title: Revisiting Inferential Benchmarks for Knowledge Graph Completion
- Authors: Shuwen Liu, Bernardo Cuenca Grau, Ian Horrocks, Egor V. Kostylev
- Abstract summary: Key feature of Machine Learning approaches for Knowledge Graph (KG) completion is their ability to learn inference patterns.
Standard completion benchmarks are not well-suited for evaluating models' abilities to learn patterns.
We propose a novel approach for designing KG completion benchmarks based on the following principles.
- Score: 29.39724559354927
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Knowledge Graph (KG) completion is the problem of extending an incomplete KG
with missing facts. A key feature of Machine Learning approaches for KG
completion is their ability to learn inference patterns, so that the predicted
facts are the results of applying these patterns to the KG. Standard completion
benchmarks, however, are not well-suited for evaluating models' abilities to
learn patterns, because the training and test sets of these benchmarks are a
random split of a given KG and hence do not capture the causality of inference
patterns. We propose a novel approach for designing KG completion benchmarks
based on the following principles: there is a set of logical rules so that the
missing facts are the results of the rules' application; the training set
includes both premises matching rule antecedents and the corresponding
conclusions; the test set consists of the results of applying the rules to the
training set; the negative examples are designed to discourage the models from
learning rules not entailed by the rule set. We use our methodology to generate
several benchmarks and evaluate a wide range of existing KG completion systems.
Our results provide novel insights on the ability of existing models to induce
inference patterns from incomplete KGs.
Related papers
- Retrieved In-Context Principles from Previous Mistakes [55.109234526031884]
In-context learning (ICL) has been instrumental in adapting Large Language Models (LLMs) to downstream tasks using correct input-output examples.
Recent advances have attempted to improve model performance through principles derived from mistakes.
We propose Retrieved In-Context Principles (RICP), a novel teacher-student framework.
arXiv Detail & Related papers (2024-07-08T07:32:26Z) - Understanding prompt engineering may not require rethinking
generalization [56.38207873589642]
We show that the discrete nature of prompts, combined with a PAC-Bayes prior given by a language model, results in generalization bounds that are remarkably tight by the standards of the literature.
This work provides a possible justification for the widespread practice of prompt engineering.
arXiv Detail & Related papers (2023-10-06T00:52:48Z) - KGxBoard: Explainable and Interactive Leaderboard for Evaluation of
Knowledge Graph Completion Models [76.01814380927507]
KGxBoard is an interactive framework for performing fine-grained evaluation on meaningful subsets of the data.
In our experiments, we highlight the findings with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.
arXiv Detail & Related papers (2022-08-23T15:11:45Z) - Knowledge Base Question Answering by Case-based Reasoning over Subgraphs [81.22050011503933]
We show that our model answers queries requiring complex reasoning patterns more effectively than existing KG completion algorithms.
The proposed model outperforms or performs competitively with state-of-the-art models on several KBQA benchmarks.
arXiv Detail & Related papers (2022-02-22T01:34:35Z) - MPLR: a novel model for multi-target learning of logical rules for
knowledge graph reasoning [5.499688003232003]
We study the problem of learning logic rules for reasoning on knowledge graphs for completing missing factual triplets.
We propose a model called MPLR that improves the existing models to fully use training data and multi-target scenarios are considered.
Experimental results empirically demonstrate that our MPLR model outperforms state-of-the-art methods on five benchmark datasets.
arXiv Detail & Related papers (2021-12-12T09:16:00Z) - Improving Knowledge Graph Representation Learning by Structure
Contextual Pre-training [9.70121995251553]
We propose a novel pre-training-then-fine-tuning framework for knowledge graph representation learning.
A KG model is pre-trained with triple classification task, followed by discriminative fine-tuning on specific downstream tasks.
Experimental results demonstrate that fine-tuning SCoP not only outperforms results of baselines on a portfolio of downstream tasks but also avoids tedious task-specific model design and parameter training.
arXiv Detail & Related papers (2021-12-08T02:50:54Z) - EngineKGI: Closed-Loop Knowledge Graph Inference [37.15381932994768]
EngineKGI is a novel closed-loop KG inference framework.
It combines KGE and rule learning to complement each other in a closed-loop pattern.
Our model outperforms other baselines on link prediction tasks.
arXiv Detail & Related papers (2021-12-02T08:02:59Z) - The MultiBERTs: BERT Reproductions for Robustness Analysis [86.29162676103385]
Re-running pretraining can lead to substantially different conclusions about performance.
We introduce MultiBERTs: a set of 25 BERT-base checkpoints.
The aim is to enable researchers to draw robust and statistically justified conclusions about pretraining procedures.
arXiv Detail & Related papers (2021-06-30T15:56:44Z) - A Hybrid Model for Learning Embeddings and Logical Rules Simultaneously
from Knowledge Graphs [20.438750956142638]
We develop a hybrid model that learns both high-quality rules and embeddings simultaneously.
Our method uses a cross feedback paradigm wherein, an embedding model is used to guide the search of a rule mining system to mine rules and infer new facts.
arXiv Detail & Related papers (2020-09-22T20:29:27Z) - Pre-training Is (Almost) All You Need: An Application to Commonsense
Reasoning [61.32992639292889]
Fine-tuning of pre-trained transformer models has become the standard approach for solving common NLP tasks.
We introduce a new scoring method that casts a plausibility ranking task in a full-text format.
We show that our method provides a much more stable training phase across random restarts.
arXiv Detail & Related papers (2020-04-29T10:54:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.