Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction

• URL: http://arxiv.org/abs/2004.01168v3
• Date: Tue, 6 Oct 2020 09:31:15 GMT
• Title: Evaluating the Calibration of Knowledge Graph Embeddings for Trustworthy Link Prediction
• Abstract summary: We investigate the trustworthiness of predictions made by knowledge graph embedding (KGE) models. We first conduct an evaluation under the standard closed-world assumption (CWA), in which predicted triples not already in the knowledge graph are considered false. We then introduce the more realistic but challenging open-world assumption (OWA), in which unobserved predictions are not considered true or false until ground-truth labels are obtained.
• Score: 16.553439385317876
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Little is known about the trustworthiness of predictions made by knowledge graph embedding (KGE) models. In this paper we take initial steps toward this direction by investigating the calibration of KGE models, or the extent to which they output confidence scores that reflect the expected correctness of predicted knowledge graph triples. We first conduct an evaluation under the standard closed-world assumption (CWA), in which predicted triples not already in the knowledge graph are considered false, and show that existing calibration techniques are effective for KGE under this common but narrow assumption. Next, we introduce the more realistic but challenging open-world assumption (OWA), in which unobserved predictions are not considered true or false until ground-truth labels are obtained. Here, we show that existing calibration techniques are much less effective under the OWA than the CWA, and provide explanations for this discrepancy. Finally, to motivate the utility of calibration for KGE from a practitioner's perspective, we conduct a unique case study of human-AI collaboration, showing that calibrated predictions can improve human performance in a knowledge graph completion task.

Related papers

• How Sharp and Bias-Robust is a Model? Dual Evaluation Perspectives on Knowledge Graph Completion [0.5271072635937927]
  Knowledge graph completion (KGC) aims to predict missing facts from the observed KG.<n>Existing metrics overlook two key perspectives for KGC evaluation.<n>We propose a novel evaluation framework (PROBE)<n> Experiments on real-world KGs reveal that existing metrics tend to over- or under-estimate the accuracy of KGC models.
  arXiv  Detail & Related papers  (2025-12-06T04:49:29Z)
• Trustworthy Prediction with Gaussian Process Knowledge Scores [7.090362431002478]
  Probabilistic models are often used to make predictions in regions of the data space where no observations are available.<n>We propose a knowledge score for predictions that quantifies the extent to which observing data have reduced our uncertainty about a prediction.<n>We demonstrate in several experiments that the knowledge score can anticipate when predictions from a GPR model are accurate.
  arXiv  Detail & Related papers  (2025-06-23T13:36:06Z)
• On the calibration of Just-in-time Defect Prediction [0.0]
  We evaluate the calibration of three JIT DP techniques to determine whether and to what extent they exhibit poor calibration. Results reveal that all evaluated JIT DP models exhibit some level of miscalibration, with ECE ranging from 2-35%.
  arXiv  Detail & Related papers  (2025-04-16T13:06:20Z)
• Quantifying the Reliability of Predictions in Detection Transformers: Object-Level Calibration and Image-Level Uncertainty [6.209833978040362]
  In practice, DETR generates hundreds of predictions that far outnumber the actual number of objects present in an image. This raises the question: can we trust and use all of these predictions? We present empirical evidence highlighting how different predictions within the same image play distinct roles, resulting in varying reliability levels.
  arXiv  Detail & Related papers  (2024-12-02T18:34:17Z)
• Selective Temporal Knowledge Graph Reasoning [70.11788354442218]
  Temporal Knowledge Graph (TKG) aims to predict future facts based on given historical ones. Existing TKG reasoning models are unable to abstain from predictions they are uncertain. We propose an abstention mechanism for TKG reasoning, which helps the existing models make selective, instead of indiscriminate, predictions.
  arXiv  Detail & Related papers  (2024-04-02T06:56:21Z)
• Counterfactual Reasoning with Knowledge Graph Embeddings [3.6311338398148534]
  Knowledge graph embeddings (KGEs) were originally developed to infer true but missing facts in incomplete knowledge repositories. In this paper, we link knowledge graph completion and counterfactual reasoning via our new task CFKGR.
  arXiv  Detail & Related papers  (2024-03-11T17:21:39Z)
• KGEx: Explaining Knowledge Graph Embeddings via Subgraph Sampling and Knowledge Distillation [6.332573781489264]
  We present KGEx, a novel method that explains individual link predictions by drawing inspiration from surrogate models research. Given a target triple to predict, KGEx trains surrogate KGE models that we use to identify important training triples. We conduct extensive experiments on two publicly available datasets, to demonstrate that KGEx is capable of providing explanations faithful to the black-box model.
  arXiv  Detail & Related papers  (2023-10-02T10:20:24Z)
• Multiclass Alignment of Confidence and Certainty for Network Calibration [10.15706847741555]
  Recent studies reveal that deep neural networks (DNNs) are prone to making overconfident predictions. We propose a new train-time calibration method, which features a simple, plug-and-play auxiliary loss known as multi-class alignment of predictive mean confidence and predictive certainty (MACC) Our method achieves state-of-the-art calibration performance for both in-domain and out-domain predictions.
  arXiv  Detail & Related papers  (2023-09-06T00:56:24Z)
• CausE: Towards Causal Knowledge Graph Embedding [13.016173217017597]
  Knowledge graph embedding (KGE) focuses on representing the entities and relations of a knowledge graph (KG) into the continuous vector spaces. We build the new paradigm of KGE in the context of causality and embedding disentanglement. We propose a Causality-enhanced knowledge graph Embedding (CausE) framework.
  arXiv  Detail & Related papers  (2023-07-21T14:25:39Z)
• T-Cal: An optimal test for the calibration of predictive models [49.11538724574202]
  We consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. We propose T-Cal, a minimax test for calibration based on a de-biased plug-in estimator of the $ell$-Expected Error (ECE)
  arXiv  Detail & Related papers  (2022-03-03T16:58:54Z)
• CAKE: A Scalable Commonsense-Aware Framework For Multi-View Knowledge Graph Completion [43.172893405453266]
  Previous knowledge graph embedding techniques suffer from invalid negative sampling and the uncertainty of fact-view link prediction. We propose a novel and scalable Commonsense-Aware Knowledge Embedding (CAKE) framework to automatically extract commonsense from factual triples with entity concepts.
  arXiv  Detail & Related papers  (2022-02-25T03:30:22Z)
• Taming Overconfident Prediction on Unlabeled Data from Hindsight [50.9088560433925]
  Minimizing prediction uncertainty on unlabeled data is a key factor to achieve good performance in semi-supervised learning. This paper proposes a dual mechanism, named ADaptive Sharpening (ADS), which first applies a soft-threshold to adaptively mask out determinate and negligible predictions. ADS significantly improves the state-of-the-art SSL methods by making it a plug-in.
  arXiv  Detail & Related papers  (2021-12-15T15:17:02Z)
• Incorporating Causal Graphical Prior Knowledge into Predictive Modeling via Simple Data Augmentation [92.96204497841032]
  Causal graphs (CGs) are compact representations of the knowledge of the data generating processes behind the data distributions. We propose a model-agnostic data augmentation method that allows us to exploit the prior knowledge of the conditional independence (CI) relations. We experimentally show that the proposed method is effective in improving the prediction accuracy, especially in the small-data regime.
  arXiv  Detail & Related papers  (2021-02-27T06:13:59Z)
• RelWalk A Latent Variable Model Approach to Knowledge Graph Embedding [50.010601631982425]
  This paper extends the random walk model (Arora et al., 2016a) of word embeddings to Knowledge Graph Embeddings (KGEs) We derive a scoring function that evaluates the strength of a relation R between two entities h (head) and t (tail) We propose a learning objective motivated by the theoretical analysis to learn KGEs from a given knowledge graph.
  arXiv  Detail & Related papers  (2021-01-25T13:31:29Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.