KGrEaT: A Framework to Evaluate Knowledge Graphs via Downstream Tasks

• URL: http://arxiv.org/abs/2308.10537v1
• Date: Mon, 21 Aug 2023 07:43:10 GMT
• Title: KGrEaT: A Framework to Evaluate Knowledge Graphs via Downstream Tasks
• Authors: Nicolas Heist, Sven Hertling, Heiko Paulheim
• Abstract summary: KGrEaT is a framework to estimate the quality of knowledge graphs via actual downstream tasks like classification, clustering, or recommendation. The framework takes a knowledge graph as input, automatically maps it to the datasets to be evaluated on, and computes performance metrics for the defined tasks.
• Score: 1.8722948221596285
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: In recent years, countless research papers have addressed the topics of knowledge graph creation, extension, or completion in order to create knowledge graphs that are larger, more correct, or more diverse. This research is typically motivated by the argumentation that using such enhanced knowledge graphs to solve downstream tasks will improve performance. Nonetheless, this is hardly ever evaluated. Instead, the predominant evaluation metrics - aiming at correctness and completeness - are undoubtedly valuable but fail to capture the complete picture, i.e., how useful the created or enhanced knowledge graph actually is. Further, the accessibility of such a knowledge graph is rarely considered (e.g., whether it contains expressive labels, descriptions, and sufficient context information to link textual mentions to the entities of the knowledge graph). To better judge how well knowledge graphs perform on actual tasks, we present KGrEaT - a framework to estimate the quality of knowledge graphs via actual downstream tasks like classification, clustering, or recommendation. Instead of comparing different methods of processing knowledge graphs with respect to a single task, the purpose of KGrEaT is to compare various knowledge graphs as such by evaluating them on a fixed task setup. The framework takes a knowledge graph as input, automatically maps it to the datasets to be evaluated on, and computes performance metrics for the defined tasks. It is built in a modular way to be easily extendable with additional tasks and datasets.

Related papers

• Class Granularity: How richly does your knowledge graph represent the real world? [0.27309692684728604]
  We propose a new metric called Open Granularity, which measures how well a knowledge graph is structured in terms of how finely classes with unique characteristics are defined. Furthermore, this research presents potential impact of Open Granularity in knowledge graph's on downstream tasks.
  arXiv  Detail & Related papers  (2024-11-10T07:57:39Z)
• G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering [61.93058781222079]
  We develop a flexible question-answering framework targeting real-world textual graphs. We introduce the first retrieval-augmented generation (RAG) approach for general textual graphs. G-Retriever performs RAG over a graph by formulating this task as a Prize-Collecting Steiner Tree optimization problem.
  arXiv  Detail & Related papers  (2024-02-12T13:13:04Z)
• State of the Art and Potentialities of Graph-level Learning [54.68482109186052]
  Graph-level learning has been applied to many tasks including comparison, regression, classification, and more. Traditional approaches to learning a set of graphs rely on hand-crafted features, such as substructures. Deep learning has helped graph-level learning adapt to the growing scale of graphs by extracting features automatically and encoding graphs into low-dimensional representations.
  arXiv  Detail & Related papers  (2023-01-14T09:15:49Z)
• Joint Language Semantic and Structure Embedding for Knowledge Graph Completion [66.15933600765835]
  We propose to jointly embed the semantics in the natural language description of the knowledge triplets with their structure information. Our method embeds knowledge graphs for the completion task via fine-tuning pre-trained language models. Our experiments on a variety of knowledge graph benchmarks have demonstrated the state-of-the-art performance of our method.
  arXiv  Detail & Related papers  (2022-09-19T02:41:02Z)
• A Survey on Visual Transfer Learning using Knowledge Graphs [0.8701566919381223]
  This survey focuses on visual transfer learning approaches using knowledge graphs (KGs) KGs can represent auxiliary knowledge either in an underlying graph-structured schema or in a vector-based knowledge graph embedding. We provide a broad overview of knowledge graph embedding methods and describe several joint training objectives suitable to combine them with high dimensional visual embeddings.
  arXiv  Detail & Related papers  (2022-01-27T20:19:55Z)
• Conditional Attention Networks for Distilling Knowledge Graphs in Recommendation [74.14009444678031]
  We propose Knowledge-aware Conditional Attention Networks (KCAN) to incorporate knowledge graph into a recommender system. We use a knowledge-aware attention propagation manner to obtain the node representation first, which captures the global semantic similarity on the user-item network and the knowledge graph. Then, by applying a conditional attention aggregation on the subgraph, we refine the knowledge graph to obtain target-specific node representations.
  arXiv  Detail & Related papers  (2021-11-03T09:40:43Z)
• GCNBoost: Artwork Classification by Label Propagation through a Knowledge Graph [32.129005474301735]
  Contextual information is often the key to structure such real world data, and we propose to use it in form of a knowledge graph. We propose a novel use of a knowledge graph, that is constructed on annotated data and pseudo-labeled data. With label propagation, we boost artwork classification by training a model using a graph convolutional network.
  arXiv  Detail & Related papers  (2021-05-25T11:50:05Z)
• Edge: Enriching Knowledge Graph Embeddings with External Text [32.01476220906261]
  We propose a knowledge graph enrichment and embedding framework named Edge. Given an original knowledge graph, we first generate a rich but noisy augmented graph using external texts in semantic and structural level. To distill the relevant knowledge and suppress the introduced noise, we design a graph alignment term in a shared embedding space between the original graph and augmented graph.
  arXiv  Detail & Related papers  (2021-04-11T03:47:06Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)
• A Survey of Adversarial Learning on Graphs [59.21341359399431]
  We investigate and summarize the existing works on graph adversarial learning tasks. Specifically, we survey and unify the existing works w.r.t. attack and defense in graph analysis tasks. We emphasize the importance of related evaluation metrics, investigate and summarize them comprehensively.
  arXiv  Detail & Related papers  (2020-03-10T12:48:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.