When Metrics Disagree: A Meta-Analysis of Knowledge-Graph-Completion Model Benchmarking

• URL: http://arxiv.org/abs/2606.10287v1
• Date: Tue, 09 Jun 2026 01:20:43 GMT
• Title: When Metrics Disagree: A Meta-Analysis of Knowledge-Graph-Completion Model Benchmarking
• Abstract summary: We present a meta-analysis of seven aggregators across five tests: consistency, cross-dataset stability, metric independence, robustness under noise, and generalizability.<n>Across tail $(h,r,?)$ and relation $(h,?,t)$ prediction, Z-score is the most balanced aggregator, which ranks DualE highest for tail prediction and FMS highest for relation prediction.<n>The framework resolves evaluation inconsistencies and offers evidence-based guidance for aggregator selection and model benchmarking in KGC.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating Knowledge Graph Completion (KGC) models remains challenging because standard assessment relies on isolated rank-based metrics such as MRR, Hits$@$k, and Mean Rank, which often produce conflicting model orderings across datasets. A model that leads on MRR may trail on Hits@1, and strong performance on one dataset may not generalize to another. This fragmentation hinders comparison, enables selective reporting, and obscures real progress. We reframe KGC evaluation as a Multi-Criteria Decision-Making (MCDM) problem and present a meta-analysis of seven aggregators across five tests: consistency, cross-dataset stability, metric independence, robustness under noise, and generalizability. Each test is averaged over leave-one-model-out (LOMO) and leave-one-group-out (LOGO) removals so that reliability reflects aggregator behavior across diverse model subsets. Across tail $(h,r,?)$ and relation $(h,?,t)$ prediction, Pareto-optimal analysis identifies Z-score as the most balanced aggregator, which ranks DualE highest for tail prediction and FMS (Flow-Modulated Scoring) highest for relation prediction. A test-sensitivity analysis using the same removals shows that consistency and stability are largely removal-invariant, while generalizability and independence are the most sensitive. The framework resolves evaluation inconsistencies and offers evidence-based guidance for aggregator selection and model benchmarking in KGC.

Related papers

• Lost in Aggregation: How Benchmarks Overlook Irreplaceable Model Strengths [23.84195042261017]
  We argue that benchmark evaluation should also consider the data-centric peak performance frontier.<n>Applying this framework to the TabArena benchmark, we find that common aggregation metrics are highly correlated.
  arXiv  Detail & Related papers  (2026-08-19T13:51:05Z)
• Back to All-Entity Ranking: Sampler-Dependent Evaluation in Continuous-Time Dynamic Graphs [0.0]
  Next-destination prediction in CTDGs ranks an observed interaction against sampled negative destinations.<n>We show that a non-uniform negative distribution changes the Bayes-optimal ranking.<n>We examine this mechanism using a factorial evaluation of repeated and new positives against seen and unseen negatives.
  arXiv  Detail & Related papers  (2026-07-30T08:37:43Z)
• DualEval: Joint Model-Item Calibration for Unified LLM Evaluation [67.99119926012686]
  DualEval is a latent model-item calibration framework that represents models and evaluation items in a shared space.<n>We apply DualEval across four domains: coding, math, miscellaneous domain-knowledge tasks, and generic everyday user queries.
  arXiv  Detail & Related papers  (2026-06-24T22:40:46Z)
• Maximum Matching Accuracy: An Instance Segmentation Evaluation Metric Utilizing Globally Optimal Matching [39.3098730337656]
  We propose a threshold-free continuous metric that finds a globally optimal one-to-one matching between predicted and ground truth objects.<n>MMA produces scores that are more stable, more sensitive, and more interpretable than existing alternatives.
  arXiv  Detail & Related papers  (2026-06-08T19:36:28Z)
• Provable Joint Decontamination for Benchmarking Multiple Large Language Models [19.929035827959822]
  We formalize benchmark decontamination as a joint selection problem and propose Joint Envelope Conformal Selection (JECS)<n>JECS computes per-model conformal p-values, aggregates them by the per-item maximum, and reconstructs a conservative envelope of the max-p null distribution.<n>Experiments across various models and benchmarks demonstrate that JECS achieves higher power than the max-p baseline while consistently maintaining the target GCR control.
  arXiv  Detail & Related papers  (2026-05-20T09:16:39Z)
• Automatic Unsupervised Ensemble Outlier Model Selection--Extended Version [16.177590208339176]
  Unsupervised outlier detection is attractive because it eliminates the need for labeled data.<n>We propose MetaEns, an unsupervised framework for selecting ensembles of outlier detection models.<n>Experiments on 39 real-world datasets show that MetaEns consistently outperforms state-of-the-art unsupervised selectors and ensemble baselines.
  arXiv  Detail & Related papers  (2026-05-15T19:15:58Z)
• IF-RewardBench: Benchmarking Judge Models for Instruction-Following Evaluation [85.56193980646981]
  We propose IF-RewardBench, a comprehensive meta-evaluation benchmark for instruction-following.<n>For each instruction, we construct a preference graph containing all pairwise preferences among multiple responses.<n>Experiments on IF-RewardBench reveal significant deficiencies in current judge models.
  arXiv  Detail & Related papers  (2026-03-05T02:21:17Z)
• STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction [78.0692157478247]
  We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning.<n>We show that STAR consistently outperforms all baselines on both score-based and rank-based metrics.
  arXiv  Detail & Related papers  (2026-02-12T16:30:07Z)
• From Global to Granular: Revealing IQA Model Performance via Correlation Surface [83.65597122328133]
  We present textbfGranularity-Modulated Correlation (GMC), which provides a structured, fine-grained analysis of IQA performance.<n>GMC includes a textbfDistribution Regulator that regularizes correlations to mitigate biases from non-uniform quality distributions.<n>Experiments on standard benchmarks show that GMC reveals performance characteristics invisible to scalar metrics, offering a more informative and reliable paradigm for analyzing, comparing, and deploying IQA models.
  arXiv  Detail & Related papers  (2026-01-29T13:55:26Z)
• Confidence and Dispersity as Signals: Unsupervised Model Evaluation and Ranking [46.95596181965493]
  This paper presents a unified and practical framework for unsupervised model evaluation and ranking.<n>We show that hybrid metrics consistently outperform single-aspect metrics on both dataset-centric and model-centric evaluation settings.
  arXiv  Detail & Related papers  (2025-10-03T12:48:11Z)
• KG-EDAS: A Meta-Metric Framework for Evaluating Knowledge Graph Completion Models [0.0]
  A major challenge in evaluating Knowledge Graphs (KGs) is comparing their performance across multiple datasets and metrics.<n>We propose KG Evaluation based on Distance from Average Solution (EDAS) to integrate multi-metric, multi-dataset performance into a unified ranking.<n>EDAS offers a global perspective that supports more informed model selection and promotes fairness in cross-dataset evaluation.
  arXiv  Detail & Related papers  (2025-08-21T08:37:35Z)
• GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models [60.48306899271866]
  We present a new framework, called GREAT Score, for global robustness evaluation of adversarial perturbation using generative models. We show high correlation and significantly reduced cost of GREAT Score when compared to the attack-based model ranking on RobustBench. GREAT Score can be used for remote auditing of privacy-sensitive black-box models.
  arXiv  Detail & Related papers  (2023-04-19T14:58:27Z)
• CARMS: Categorical-Antithetic-REINFORCE Multi-Sample Gradient Estimator [60.799183326613395]
  We propose an unbiased estimator for categorical random variables based on multiple mutually negatively correlated (jointly antithetic) samples. CARMS combines REINFORCE with copula based sampling to avoid duplicate samples and reduce its variance, while keeping the estimator unbiased using importance sampling. We evaluate CARMS on several benchmark datasets on a generative modeling task, as well as a structured output prediction task, and find it to outperform competing methods including a strong self-control baseline.
  arXiv  Detail & Related papers  (2021-10-26T20:14:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.