How Sharp and Bias-Robust is a Model? Dual Evaluation Perspectives on Knowledge Graph Completion

• URL: http://arxiv.org/abs/2512.06296v1
• Date: Sat, 06 Dec 2025 04:49:29 GMT
• Title: How Sharp and Bias-Robust is a Model? Dual Evaluation Perspectives on Knowledge Graph Completion
• Abstract summary: Knowledge graph completion (KGC) aims to predict missing facts from the observed KG.<n>Existing metrics overlook two key perspectives for KGC evaluation.<n>We propose a novel evaluation framework (PROBE)<n> Experiments on real-world KGs reveal that existing metrics tend to over- or under-estimate the accuracy of KGC models.
• Score: 0.5271072635937927
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge graph completion (KGC) aims to predict missing facts from the observed KG. While a number of KGC models have been studied, the evaluation of KGC still remain underexplored. In this paper, we observe that existing metrics overlook two key perspectives for KGC evaluation: (A1) predictive sharpness -- the degree of strictness in evaluating an individual prediction, and (A2) popularity-bias robustness -- the ability to predict low-popularity entities. Toward reflecting both perspectives, we propose a novel evaluation framework (PROBE), which consists of a rank transformer (RT) estimating the score of each prediction based on a required level of predictive sharpness and a rank aggregator (RA) aggregating all the scores in a popularity-aware manner. Experiments on real-world KGs reveal that existing metrics tend to over- or under-estimate the accuracy of KGC models, whereas PROBE yields a comprehensive understanding of KGC models and reliable evaluation results.

Related papers

• PROBE-Web: An Interactive System for Probing Evaluation Landscapes of Knowledge Graph Completion Models [0.5271072635937927]
  PROBE-Web is an interactive system for probing diverse evaluation landscapes for KGC models.<n>It enables users to evaluate flexibly KGC models by adjusting two critical perspectives: (P1) predictive sharpness and (P2) popularity-bias robustness.
  arXiv  Detail & Related papers  (2026-06-08T02:08:23Z)
• Generalized Rank-based Evaluation for Knowledge Graph Completion: Perspectives, Framework, and Analyses [2.7836206220202757]
  We introduce two important perspectives for KGC evaluation that are overlooked by existing evaluation metrics.<n>We propose a generalized evaluation framework, PROBE, that estimates the score of each prediction based on a desired level of predictive sharpness.<n>We show that PROBE better maintains such consistency, providing a more reliable estimate of intrinsic model performance than existing metrics.
  arXiv  Detail & Related papers  (2026-06-08T02:00:38Z)
• When to Trust: A Causality-Aware Calibration Framework for Accurate Knowledge Graph Retrieval-Augmented Generation [9.05691591939794]
  Knowledge Graph Retrieval-Augmented Generation (KG-RAG) extends the RAG paradigm by incorporating structured knowledge from knowledge graphs.<n>Existing KG-RAG models are often severely overconfident, producing high-confidence predictions even when retrieved sub-graphs are incomplete or unreliable.<n>We propose Ca2KG, a Causality-aware framework for KG-RAG, which integrates counterfactual prompting, with a panel-based re-scoring mechanism.
  arXiv  Detail & Related papers  (2026-01-14T07:22:59Z)
• Concept Regions Matter: Benchmarking CLIP with a New Cluster-Importance Approach [20.898059440239603]
  Cluster-based Concept Importance (CCI) is a novel interpretability method.<n>CCI sets a new state of the art on faithfulness benchmarks.<n>We present a comprehensive evaluation of eighteen CLIP variants.
  arXiv  Detail & Related papers  (2025-11-17T05:01:24Z)
• Evaluating Cumulative Spectral Gradient as a Complexity Measure [0.0]
  Cumulative Spectral Gradient (CSG) was proposed as a dataset complexity measure.<n>In this work, we rigorously assess CSG behavior on standard knowledge graph link prediction benchmarks.
  arXiv  Detail & Related papers  (2025-09-02T15:10:25Z)
• CompassJudger-2: Towards Generalist Judge Model via Verifiable Rewards [72.44810390478229]
  CompassJudger-2 is a novel generalist judge model that overcomes limitations via a task-driven, multi-domain data curation strategy.<n> CompassJudger-2 achieves superior results across multiple judge and reward benchmarks.
  arXiv  Detail & Related papers  (2025-07-12T01:34:24Z)
• On Large-scale Evaluation of Embedding Models for Knowledge Graph Completion [1.2703808802607108]
  Knowledge graph embedding (KGE) models are extensively studied for knowledge graph completion.<n>Standard evaluation metrics rely on the closed-world assumption, which penalizes models for correctly predicting missing triples.<n>This paper conducts a comprehensive evaluation of four representative KGE models on large-scale datasets FB-CVT-REV and FB+CVT-REV.
  arXiv  Detail & Related papers  (2025-04-11T20:49:02Z)
• Overcoming Pitfalls in Graph Contrastive Learning Evaluation: Toward Comprehensive Benchmarks [60.82579717007963]
  We introduce an enhanced evaluation framework designed to more accurately gauge the effectiveness, consistency, and overall capability of Graph Contrastive Learning (GCL) methods.
  arXiv  Detail & Related papers  (2024-02-24T01:47:56Z)
• Memory Consistency Guided Divide-and-Conquer Learning for Generalized Category Discovery [56.172872410834664]
  Generalized category discovery (GCD) aims at addressing a more realistic and challenging setting of semi-supervised learning. We propose a Memory Consistency guided Divide-and-conquer Learning framework (MCDL) Our method outperforms state-of-the-art models by a large margin on both seen and unseen classes of the generic image recognition.
  arXiv  Detail & Related papers  (2024-01-24T09:39:45Z)
• GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models [60.48306899271866]
  We present a new framework, called GREAT Score, for global robustness evaluation of adversarial perturbation using generative models. We show high correlation and significantly reduced cost of GREAT Score when compared to the attack-based model ranking on RobustBench. GREAT Score can be used for remote auditing of privacy-sensitive black-box models.
  arXiv  Detail & Related papers  (2023-04-19T14:58:27Z)
• Evaluating Probabilistic Classifiers: The Triptych [62.997667081978825]
  We propose and study a triptych of diagnostic graphics that focus on distinct and complementary aspects of forecast performance. The reliability diagram addresses calibration, the receiver operating characteristic (ROC) curve diagnoses discrimination ability, and the Murphy diagram visualizes overall predictive performance and value.
  arXiv  Detail & Related papers  (2023-01-25T19:35:23Z)
• KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models [76.01814380927507]
  KGxBoard is an interactive framework for performing fine-grained evaluation on meaningful subsets of the data. In our experiments, we highlight the findings with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.
  arXiv  Detail & Related papers  (2022-08-23T15:11:45Z)
• Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction [16.553439385317876]
  We investigate the trustworthiness of predictions made by knowledge graph embedding (KGE) models. We first conduct an evaluation under the standard closed-world assumption (CWA), in which predicted triples not already in the knowledge graph are considered false. We then introduce the more realistic but challenging open-world assumption (OWA), in which unobserved predictions are not considered true or false until ground-truth labels are obtained.
  arXiv  Detail & Related papers  (2020-04-02T17:46:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.