KGrEaT: A Framework to Evaluate Knowledge Graphs via Downstream Tasks
- URL: http://arxiv.org/abs/2308.10537v1
- Date: Mon, 21 Aug 2023 07:43:10 GMT
- Title: KGrEaT: A Framework to Evaluate Knowledge Graphs via Downstream Tasks
- Authors: Nicolas Heist, Sven Hertling, Heiko Paulheim
- Abstract summary: KGrEaT is a framework to estimate the quality of knowledge graphs via actual downstream tasks like classification, clustering, or recommendation.
The framework takes a knowledge graph as input, automatically maps it to the datasets to be evaluated on, and computes performance metrics for the defined tasks.
- Score: 1.8722948221596285
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: In recent years, countless research papers have addressed the topics of
knowledge graph creation, extension, or completion in order to create knowledge
graphs that are larger, more correct, or more diverse. This research is
typically motivated by the argumentation that using such enhanced knowledge
graphs to solve downstream tasks will improve performance. Nonetheless, this is
hardly ever evaluated. Instead, the predominant evaluation metrics - aiming at
correctness and completeness - are undoubtedly valuable but fail to capture the
complete picture, i.e., how useful the created or enhanced knowledge graph
actually is. Further, the accessibility of such a knowledge graph is rarely
considered (e.g., whether it contains expressive labels, descriptions, and
sufficient context information to link textual mentions to the entities of the
knowledge graph). To better judge how well knowledge graphs perform on actual
tasks, we present KGrEaT - a framework to estimate the quality of knowledge
graphs via actual downstream tasks like classification, clustering, or
recommendation. Instead of comparing different methods of processing knowledge
graphs with respect to a single task, the purpose of KGrEaT is to compare
various knowledge graphs as such by evaluating them on a fixed task setup. The
framework takes a knowledge graph as input, automatically maps it to the
datasets to be evaluated on, and computes performance metrics for the defined
tasks. It is built in a modular way to be easily extendable with additional
tasks and datasets.
Related papers
- G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering [61.93058781222079]
We develop a flexible question-answering framework targeting real-world textual graphs.
We introduce the first retrieval-augmented generation (RAG) approach for general textual graphs.
G-Retriever performs RAG over a graph by formulating this task as a Prize-Collecting Steiner Tree optimization problem.
arXiv Detail & Related papers (2024-02-12T13:13:04Z) - State of the Art and Potentialities of Graph-level Learning [54.68482109186052]
Graph-level learning has been applied to many tasks including comparison, regression, classification, and more.
Traditional approaches to learning a set of graphs rely on hand-crafted features, such as substructures.
Deep learning has helped graph-level learning adapt to the growing scale of graphs by extracting features automatically and encoding graphs into low-dimensional representations.
arXiv Detail & Related papers (2023-01-14T09:15:49Z) - Joint Language Semantic and Structure Embedding for Knowledge Graph
Completion [66.15933600765835]
We propose to jointly embed the semantics in the natural language description of the knowledge triplets with their structure information.
Our method embeds knowledge graphs for the completion task via fine-tuning pre-trained language models.
Our experiments on a variety of knowledge graph benchmarks have demonstrated the state-of-the-art performance of our method.
arXiv Detail & Related papers (2022-09-19T02:41:02Z) - A Survey on Visual Transfer Learning using Knowledge Graphs [0.8701566919381223]
This survey focuses on visual transfer learning approaches using knowledge graphs (KGs)
KGs can represent auxiliary knowledge either in an underlying graph-structured schema or in a vector-based knowledge graph embedding.
We provide a broad overview of knowledge graph embedding methods and describe several joint training objectives suitable to combine them with high dimensional visual embeddings.
arXiv Detail & Related papers (2022-01-27T20:19:55Z) - Conditional Attention Networks for Distilling Knowledge Graphs in
Recommendation [74.14009444678031]
We propose Knowledge-aware Conditional Attention Networks (KCAN) to incorporate knowledge graph into a recommender system.
We use a knowledge-aware attention propagation manner to obtain the node representation first, which captures the global semantic similarity on the user-item network and the knowledge graph.
Then, by applying a conditional attention aggregation on the subgraph, we refine the knowledge graph to obtain target-specific node representations.
arXiv Detail & Related papers (2021-11-03T09:40:43Z) - GCNBoost: Artwork Classification by Label Propagation through a
Knowledge Graph [32.129005474301735]
Contextual information is often the key to structure such real world data, and we propose to use it in form of a knowledge graph.
We propose a novel use of a knowledge graph, that is constructed on annotated data and pseudo-labeled data.
With label propagation, we boost artwork classification by training a model using a graph convolutional network.
arXiv Detail & Related papers (2021-05-25T11:50:05Z) - Edge: Enriching Knowledge Graph Embeddings with External Text [32.01476220906261]
We propose a knowledge graph enrichment and embedding framework named Edge.
Given an original knowledge graph, we first generate a rich but noisy augmented graph using external texts in semantic and structural level.
To distill the relevant knowledge and suppress the introduced noise, we design a graph alignment term in a shared embedding space between the original graph and augmented graph.
arXiv Detail & Related papers (2021-04-11T03:47:06Z) - Exploiting Structured Knowledge in Text via Graph-Guided Representation
Learning [73.0598186896953]
We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs.
Building upon entity-level masked language models, our first contribution is an entity masking scheme.
In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
arXiv Detail & Related papers (2020-04-29T14:22:42Z) - A Survey of Adversarial Learning on Graphs [59.21341359399431]
We investigate and summarize the existing works on graph adversarial learning tasks.
Specifically, we survey and unify the existing works w.r.t. attack and defense in graph analysis tasks.
We emphasize the importance of related evaluation metrics, investigate and summarize them comprehensively.
arXiv Detail & Related papers (2020-03-10T12:48:00Z) - Knowledge Graphs [43.06435841693428]
We motivate and contrast various graph-based data models and query languages that are used for knowledge graphs.
We explain how knowledge can be represented and extracted using a combination of deductive and inductive techniques.
We conclude with high-level future research directions for knowledge graphs.
arXiv Detail & Related papers (2020-03-04T20:20:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.