KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text

• URL: http://arxiv.org/abs/2607.10212v1
• Date: Sat, 11 Jul 2026 08:45:15 GMT
• Title: KGCQual: An Interpretable Framework for Evaluating the Knowledge Graph Construction Quality from Text
• Abstract summary: We propose a novel, interpretable metric for intrinsic Knowledge Graph quality assessment.<n>It measures how closely an automatically extracted graph approximates an "ideal" graph.<n>We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets.
• Score: 6.373904244962351
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Knowledge Graphs (KGs) are increasingly constructed through automated extraction pipelines; however, such systems often introduce spurious or incomplete triples, which degrade downstream performance. Existing evaluation practices rely heavily on task-specific metrics or small-scale manual verification, offering limited insight into the structural and semantic fidelity of extracted graphs. We propose a novel, interpretable metric for intrinsic KG quality assessment that measures how closely an automatically extracted graph approximates an "ideal" graph capturing the key noun phrases, predicate relations, and basic linguistic phenomena such as negation expressed in the source text. Our framework integrates two complementary components: (1) an entity-level assessment that evaluates completeness, resolution quality, and connectivity, and (2) a relation-level assessment that judges predicate preservation and multiplicity using lexical similarity, dependency-parse alignment, and light-weight negation handling to ensure semantic faithfulness. We evaluate our metric across multiple state-of-the-art triple extraction systems and datasets, including WebNLG, TinyButMighty, and BenchIE, demonstrating that it reliably identifies omissions, redundancy, and structural deviations that existing metrics overlook. Our work offers a scalable, model-agnostic, and interpretable framework for comparing automated KG construction methods and provides a foundation for standardised evaluation. We further validate the metric through an ablation study isolating noun and verb components, and a downstream evaluation showing that KGCQual scores correlate significantly with link prediction performance on the same extracted KGs. The code repository is available at https://github.com/kracr/kg-quality-metric.

Related papers

• Evaluation of Contextual Understanding in Large Language Models [0.8150815269284862]
  Large Language Models (LLMs) demonstrate impressive performance across diverse NLP tasks, yet their ability to exhibit genuine contextual understanding remains uncertain.<n>Traditional evaluation metrics such as perplexity, BiLingual Evaluation Understudy (BLEU), or surface-level accuracy fail to reveal how well LLMs extract, integrate, and reason over contextual information.<n>We propose a novel knowledge graph-based evaluation framework introducing Semantic Structural Similarity for KGs (S3KG)
  arXiv  Detail & Related papers  (2026-09-08T16:41:50Z)
• Q-GNN: Query-Conditioned Graph Neural Networks with Type Awareness for Knowledge Graph Completion [38.265175718191735]
  Knowledge Graph Completion aims at predicting missing triplets from incomplete knowledge graphs.<n>Recent Graph Neural Network (GNN)-based methods have achieved remarkable success by performing message passing over query-centered local subgraphs.
  arXiv  Detail & Related papers  (2026-06-04T03:07:22Z)
• SKG-Eval: Stateful Evaluation of Multi-Turn Dialogue via Incremental Semantic Knowledge Graphs [0.0]
  SKG-Eval is a quasi-deterministic and interpretable framework that models dialogue as an evolving Semantic Knowledge Graph.<n>We show that SKG-Eval achieves higher correlation with human judgments and substantially improves detection of long-range inconsistencies in extended conversations.
  arXiv  Detail & Related papers  (2026-05-15T21:39:48Z)
• A Unified Benchmark for Evaluating Knowledge Graph Construction Methods and Graph Neural Networks [1.9307742024764771]
  We introduce a dual-purpose benchmark designed to jointly evaluate the performance of Graph Neural Networks (GNNs) on noisy, text-derived graphs.<n>The benchmark is built in the biomedical domain from a single textual corpus.
  arXiv  Detail & Related papers  (2026-05-06T21:53:49Z)
• Enrich-on-Graph: Query-Graph Alignment for Complex Reasoning with LLM Enriching [61.824094419641575]
  Large Language Models (LLMs) struggle with hallucinations and factual errors in knowledge-intensive scenarios like knowledge graph question answering (KGQA)<n>We attribute this to the semantic gap between structured knowledge graphs (KGs) and unstructured queries, caused by inherent differences in their focuses and structures.<n>Existing methods usually employ resource-intensive, non-scalable reasoning on vanilla KGs, but overlook this gap.<n>We propose a flexible framework, Enrich-on-Graph (EoG), which leverages LLMs' prior knowledge to enrich KGs, bridge the semantic gap between graphs and queries.
  arXiv  Detail & Related papers  (2025-09-25T06:48:52Z)
• Towards Explainable Job Title Matching: Leveraging Semantic Textual Relatedness and Knowledge Graphs [0.19116784879310025]
  This study investigates semantic textual relatedness (STR) in the context of job title matching.<n>We introduce a self-supervised hybrid architecture that combines dense sentence embeddings with domain-specific Knowledge Graphs.<n>We show that fine-tuned SBERT models augmented with KGs produce consistent improvements in the high-STR region.
  arXiv  Detail & Related papers  (2025-09-11T15:02:54Z)
• Understanding the Effect of Knowledge Graph Extraction Error on Downstream Graph Analyses: A Case Study on Affiliation Graphs [4.818309069556584]
  Knowledge graphs (KGs) are useful for analyzing social structures, community dynamics, institutional memberships, and other complex relationships.<n>Recent advances in large language models (LLMs) have improved the scalability and accessibility of automated KG extraction from large text corpora.<n>The impacts of extraction errors on downstream analyses are poorly understood, especially for applied scientists who depend on accurate KGs for real-world insights.
  arXiv  Detail & Related papers  (2025-06-14T06:14:06Z)
• Cobra Effect in Reference-Free Image Captioning Metrics [58.438648377314436]
  A proliferation of reference-free methods, leveraging visual-language pre-trained models (VLMs), has emerged. In this paper, we study if there are any deficiencies in reference-free metrics. We employ GPT-4V as an evaluative tool to assess generated sentences and the result reveals that our approach achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2024-02-18T12:36:23Z)
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion [51.126166442122546]
  We introduce the Contextualization Distillation strategy, a plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models to transform compact, structural triplets into context-rich segments. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach.
  arXiv  Detail & Related papers  (2024-01-28T08:56:49Z)
• Towards Interpretable and Efficient Automatic Reference-Based Summarization Evaluation [160.07938471250048]
  Interpretability and efficiency are two important considerations for the adoption of neural automatic metrics. We develop strong-performing automatic metrics for reference-based summarization evaluation.
  arXiv  Detail & Related papers  (2023-03-07T02:49:50Z)
• GO FIGURE: A Meta Evaluation of Factuality in Summarization [131.1087461486504]
  We introduce GO FIGURE, a meta-evaluation framework for evaluating factuality evaluation metrics. Our benchmark analysis on ten factuality metrics reveals that our framework provides a robust and efficient evaluation. It also reveals that while QA metrics generally improve over standard metrics that measure factuality across domains, performance is highly dependent on the way in which questions are generated.
  arXiv  Detail & Related papers  (2020-10-24T08:30:20Z)
• Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary [65.37544133256499]
  We propose a metric to evaluate the content quality of a summary using question-answering (QA) We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval.
  arXiv  Detail & Related papers  (2020-10-01T15:33:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.