KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models

• URL: http://arxiv.org/abs/2208.11024v1
• Date: Tue, 23 Aug 2022 15:11:45 GMT
• Title: KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models
• Authors: Haris Widjaja, Kiril Gashteovski, Wiem Ben Rim, Pengfei Liu, Christopher Malon, Daniel Ruffinelli, Carolin Lawrence, Graham Neubig
• Abstract summary: KGxBoard is an interactive framework for performing fine-grained evaluation on meaningful subsets of the data. In our experiments, we highlight the findings with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.
• Score: 76.01814380927507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge Graphs (KGs) store information in the form of (head, predicate, tail)-triples. To augment KGs with new knowledge, researchers proposed models for KG Completion (KGC) tasks such as link prediction; i.e., answering (h; p; ?) or (?; p; t) queries. Such models are usually evaluated with averaged metrics on a held-out test set. While useful for tracking progress, averaged single-score metrics cannot reveal what exactly a model has learned -- or failed to learn. To address this issue, we propose KGxBoard: an interactive framework for performing fine-grained evaluation on meaningful subsets of the data, each of which tests individual and interpretable capabilities of a KGC model. In our experiments, we highlight the findings that we discovered with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.

Related papers

• Exploiting Large Language Models Capabilities for Question Answer-Driven Knowledge Graph Completion Across Static and Temporal Domains [8.472388165833292]
  This paper introduces a new generative completion framework called Generative Subgraph-based KGC (GS-KGC) GS-KGC employs a question-answering format to directly generate target entities, addressing the challenge of questions having multiple possible answers. Our method generates negative samples using known facts to facilitate the discovery of new information.
  arXiv  Detail & Related papers  (2024-08-20T13:13:41Z)
• Knowledge Graph Completion using Structural and Textual Embeddings [0.0]
  We propose a relations prediction model that harnesses both textual and structural information within Knowledge Graphs. Our approach integrates walks-based embeddings with language model embeddings to effectively represent nodes. We demonstrate that our model achieves competitive results in the relation prediction task when evaluated on a widely used dataset.
  arXiv  Detail & Related papers  (2024-04-24T21:04:14Z)
• Revisiting Inferential Benchmarks for Knowledge Graph Completion [29.39724559354927]
  Key feature of Machine Learning approaches for Knowledge Graph (KG) completion is their ability to learn inference patterns. Standard completion benchmarks are not well-suited for evaluating models' abilities to learn patterns. We propose a novel approach for designing KG completion benchmarks based on the following principles.
  arXiv  Detail & Related papers  (2023-06-07T22:35:39Z)
• GreenKGC: A Lightweight Knowledge Graph Completion Method [32.528770408502396]
  GreenKGC aims to discover missing relationships between entities in knowledge graphs. It consists of three modules: representation learning, feature pruning, and decision learning. In low dimensions, GreenKGC can outperform SOTA methods in most datasets.
  arXiv  Detail & Related papers  (2022-08-19T03:33:45Z)
• A Graph-Enhanced Click Model for Web Search [67.27218481132185]
  We propose a novel graph-enhanced click model (GraphCM) for web search. We exploit both intra-session and inter-session information for the sparsity and cold-start problems.
  arXiv  Detail & Related papers  (2022-06-17T08:32:43Z)
• Sequence-to-Sequence Knowledge Graph Completion and Question Answering [8.207403859762044]
  We show that an off-the-shelf encoder-decoder Transformer model can serve as a scalable and versatile KGE model. We achieve this by posing KG link prediction as a sequence-to-sequence task and exchange the triple scoring approach taken by prior KGE methods with autoregressive decoding.
  arXiv  Detail & Related papers  (2022-03-19T13:01:49Z)
• Knowledge Graph Question Answering Leaderboard: A Community Resource to Prevent a Replication Crisis [61.740077541531726]
  We provide a new central and open leaderboard for any KGQA benchmark dataset as a focal point for the community. Our analysis highlights existing problems during the evaluation of KGQA systems.
  arXiv  Detail & Related papers  (2022-01-20T13:46:01Z)
• Learning Intents behind Interactions with Knowledge Graph for Recommendation [93.08709357435991]
  Knowledge graph (KG) plays an increasingly important role in recommender systems. Existing GNN-based models fail to identify user-item relation at a fine-grained level of intents. We propose a new model, Knowledge Graph-based Intent Network (KGIN)
  arXiv  Detail & Related papers  (2021-02-14T03:21:36Z)
• KACC: A Multi-task Benchmark for Knowledge Abstraction, Concretization and Completion [99.47414073164656]
  A comprehensive knowledge graph (KG) contains an instance-level entity graph and an ontology-level concept graph. The two-view KG provides a testbed for models to "simulate" human's abilities on knowledge abstraction, concretization, and completion. We propose a unified KG benchmark by improving existing benchmarks in terms of dataset scale, task coverage, and difficulty.
  arXiv  Detail & Related papers  (2020-04-28T16:21:57Z)
• Toward Subgraph-Guided Knowledge Graph Question Generation with Graph Neural Networks [53.58077686470096]
  Knowledge graph (KG) question generation (QG) aims to generate natural language questions from KGs and target answers. In this work, we focus on a more realistic setting where we aim to generate questions from a KG subgraph and target answers.
  arXiv  Detail & Related papers  (2020-04-13T15:43:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.