Link Prediction or Perdition: the Seeds of Instability in Knowledge Graph Embeddings

• URL: http://arxiv.org/abs/2606.03365v1
• Date: Tue, 02 Jun 2026 09:13:50 GMT
• Title: Link Prediction or Perdition: the Seeds of Instability in Knowledge Graph Embeddings
• Abstract summary: Embedding models (KGEMs) constitute the main link prediction approach to complete knowledge graphs.<n>Standard evaluation protocols emphasize rank-based metrics such as MRR or Hits@$K$, but usually overlook the influence of random seeds on result stability.<n>We find that high-performance models actually produce divergent predictions at the triple level and highly variable spaces.
• Score: 2.1174215880331766
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Embedding models (KGEMs) constitute the main link prediction approach to complete knowledge graphs. Standard evaluation protocols emphasize rank-based metrics such as MRR or Hits@$K$, but usually overlook the influence of random seeds on result stability. Moreover, these metrics conceal potential instabilities in individual predictions and in the organization of embedding spaces. In this work, we conduct a systematic stability analysis of multiple KGEMs across several datasets. We find that high-performance models actually produce divergent predictions at the triple level and highly variable embedding spaces. By isolating stochastic factors (i.e., initialization, triple ordering, negative sampling, dropout, hardware), we show that each independently induces instability of comparable magnitude. Furthermore, for a given model, hyperparameter configurations with better MRR are not guaranteed to be more stable. Moreover, voting, albeit a known remediation mechanism, only provides a limited enhancement of stability. These findings highlight critical limitations of current benchmarking protocols, and raise concerns about the reliability of KGEMs for knowledge graph completion.

Related papers

• TRIE: An Evaluation Framework for Stochastic PDE Surrogates [5.2492759201709065]
  We introduce TRIE, an evaluation framework for PDE surrogates.<n>We ask whether models reproduce invariant measures, provide trustworthy predictive uncertainty, and scale to efficient generation.<n>We show that latent generative models with automatic dimension discovery retain much of this statistical fidelity while reducing Kolmogorov inference time by roughly $12times$.
  arXiv  Detail & Related papers  (2026-06-30T21:20:37Z)
• When Do Autoregressive Sequence Models Forecast Physical Wavefields? A Controlled Study on Synthetic Seismograms [43.8784307709823]
  Long-horizon autoregressive forecasting of physical signals is limited by error accumulation.<n>We ask when such rollout stays stable using synthetic three-component seismograms as a physically structured testbed.
  arXiv  Detail & Related papers  (2026-06-09T13:46:14Z)
• Conformal Reliability: A New Evaluation Metric for Conditional Generation [62.761166941396844]
  Conditional generative models have recently remarkable success in various applications.<n>We propose a novel evaluation metric called reliability based on prediction set at a pre-specified confidence level.<n>We introduce Conformal ReLiability (CReL), a framework that can (i) construct a prediction set and (ii) accurately optimize the reliability score within the constructed prediction set.
  arXiv  Detail & Related papers  (2026-05-29T03:52:44Z)
• Is Capability a Liability? More Capable Language Models Make Worse Forecasts When It Matters Most [11.4807141145402]
  In inverse scaling, more capable models produce worse distributional forecasts on forecasting problems with superlinear growth and tail risk of regime change.<n>The pattern appears on ForecastBench-Sim, in forecasting synthetic SIR epidemics with a matched linear control, and replicates in real-world datasets on COVID-19, measles, housing markets, and hyperinflation.<n>This inverse scaling does not appear on single-threshold metrics common in LLM forecasting benchmarks, reversing the sign of the capability--accuracy relationship on identical outputs.
  arXiv  Detail & Related papers  (2026-05-21T16:14:33Z)
• Multimodal Learning on Low-Quality Data with Conformal Predictive Self-Calibration [72.0672328514289]
  Multimodal learning often grapples with the challenge of low-quality data.<n>We propose a unified framework, termed Conformal Predictive Self-Calibration.<n>Our framework consistently outperforms existing state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-05T14:48:52Z)
• Feature Attribution Stability Suite: How Stable Are Post-Hoc Attributions? [0.0]
  Post-hoc feature attribution methods are widely deployed in safety-critical vision systems.<n>We introduce the Feature Stability Suite (FASS), a benchmark that enforces prediction-invariance filtering.<n>FASS decomposes stability into three complementary metrics: structural similarity, rank correlation, and top-k Jaccard overlap.
  arXiv  Detail & Related papers  (2026-04-02T21:32:54Z)
• Beyond the Mean: Distribution-Aware Loss Functions for Bimodal Regression [13.50774354486614]
  We propose a family of distribution-aware loss functions integrating normalized RMSE with Wasserstein and Cramér distances.<n>Our framework strictly dominates MDNs in both fidelity and robustness, offering a reliable tool for aleatoric uncertainty estimation in trustworthy AI systems.
  arXiv  Detail & Related papers  (2026-03-20T18:37:05Z)
• STAR : Bridging Statistical and Agentic Reasoning for Large Model Performance Prediction [78.0692157478247]
  We propose STAR, a framework that bridges data-driven STatistical expectations with knowledge-driven Agentic Reasoning.<n>We show that STAR consistently outperforms all baselines on both score-based and rank-based metrics.
  arXiv  Detail & Related papers  (2026-02-12T16:30:07Z)
• Revisiting Multivariate Time Series Forecasting with Missing Values [65.30332997607141]
  Missing values are common in real-world time series.<n>Current approaches have developed an imputation-then-prediction framework that uses imputation modules to fill in missing values, followed by forecasting on the imputed data.<n>This framework overlooks a critical issue: there is no ground truth for the missing values, making the imputation process susceptible to errors that can degrade prediction accuracy.<n>We introduce Consistency-Regularized Information Bottleneck (CRIB), a novel framework built on the Information Bottleneck principle.
  arXiv  Detail & Related papers  (2025-09-27T20:57:48Z)
• Probabilistic Scores of Classifiers, Calibration is not Enough [0.32985979395737786]
  In binary classification tasks, accurate representation of probabilistic predictions is essential for various real-world applications. In this study, we highlight approaches that prioritize the alignment between predicted scores and true probability distributions. Our findings reveal limitations in traditional calibration metrics, which could undermine the reliability of predictive models for critical decision-making.
  arXiv  Detail & Related papers  (2024-08-06T19:53:00Z)
• Evaluating Probabilistic Classifiers: The Triptych [62.997667081978825]
  We propose and study a triptych of diagnostic graphics that focus on distinct and complementary aspects of forecast performance. The reliability diagram addresses calibration, the receiver operating characteristic (ROC) curve diagnoses discrimination ability, and the Murphy diagram visualizes overall predictive performance and value.
  arXiv  Detail & Related papers  (2023-01-25T19:35:23Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.