Conformalized Answer Set Prediction for Knowledge Graph Embedding

• URL: http://arxiv.org/abs/2408.08248v3
• Date: Sat, 25 Jan 2025 17:44:05 GMT
• Title: Conformalized Answer Set Prediction for Knowledge Graph Embedding
• Authors: Yuqicheng Zhu, Nico Potyka, Jiarong Pan, Bo Xiong, Yunjie He, Evgeny Kharlamov, Steffen Staab,
• Abstract summary: We show how conformal prediction can be used to generate correct answer sets for link prediction tasks.<n>We also demonstrate that the generated answer sets often have a sensible size and that the size adapts well with respect to the difficulty of the query.
• Score: 26.803481853670064
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Knowledge graph embeddings (KGE) apply machine learning methods on knowledge graphs (KGs) to provide non-classical reasoning capabilities based on similarities and analogies. The learned KG embeddings are typically used to answer queries by ranking all potential answers, but rankings often lack a meaningful probabilistic interpretation - lower-ranked answers do not necessarily have a lower probability of being true. This limitation makes it difficult to quantify uncertainty of model's predictions, posing challenges for the application of KGE methods in high-stakes domains like medicine. We address this issue by applying the theory of conformal prediction that allows generating answer sets, which contain the correct answer with probabilistic guarantees. We explain how conformal prediction can be used to generate such answer sets for link prediction tasks. Our empirical evaluation on four benchmark datasets using six representative KGE methods validates that the generated answer sets satisfy the probabilistic guarantees given by the theory of conformal prediction. We also demonstrate that the generated answer sets often have a sensible size and that the size adapts well with respect to the difficulty of the query.

Related papers

• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Predicate-Conditional Conformalized Answer Sets for Knowledge Graph Embeddings [27.138935426612306]
  CondKGCP approximates predicate-conditional coverage guarantees while maintaining compact prediction sets.<n>We prove the theoretical guarantees and demonstrate empirical effectiveness of CondKGCP by comprehensive evaluations.
  arXiv  Detail & Related papers  (2025-05-22T16:33:20Z)
• E-Values Expand the Scope of Conformal Prediction [49.1574468325115]
  Conformal prediction is a powerful framework for distribution-free uncertainty quantification. In this paper, we explore an alternative approach based on e-values, known as conformal e-prediction. E-values offer key advantages that cannot be achieved with p-values, enabling new theoretical and practical capabilities.
  arXiv  Detail & Related papers  (2025-03-17T10:54:30Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples. We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Predictive Multiplicity of Knowledge Graph Embeddings in Link Prediction [31.79167587393948]
  Knowledge graph embedding (KGE) models are often used to predict missing links for knowledge graphs (KGs) Multiple KG embeddings can perform almost equally well for link prediction yet give conflicting predictions for unseen queries. It poses substantial risks for KGE-based applications in high-stake domains but has been overlooked in KGE research.
  arXiv  Detail & Related papers  (2024-08-15T15:54:02Z)
• Conformal Predictions for Probabilistically Robust Scalable Machine Learning Classification [1.757077789361314]
  Conformal predictions make it possible to define reliable and robust learning algorithms. They are essentially a method for evaluating whether an algorithm is good enough to be used in practice. This paper defines a reliable learning framework for classification from the very beginning of its design.
  arXiv  Detail & Related papers  (2024-03-15T14:59:24Z)
• Less is More: One-shot Subgraph Reasoning on Large-scale Knowledge Graphs [49.547988001231424]
  We propose the one-shot-subgraph link prediction to achieve efficient and adaptive prediction. Design principle is that, instead of directly acting on the whole KG, the prediction procedure is decoupled into two steps. We achieve promoted efficiency and leading performances on five large-scale benchmarks.
  arXiv  Detail & Related papers  (2024-03-15T12:00:12Z)
• Calibrating AI Models for Wireless Communications via Conformal Prediction [55.47458839587949]
  Conformal prediction is applied for the first time to the design of AI for communication systems. This paper investigates the application of conformal prediction as a general framework to obtain AI models that produce decisions with formal calibration guarantees.
  arXiv  Detail & Related papers  (2022-12-15T12:52:23Z)
• Practical Adversarial Multivalid Conformal Prediction [27.179891682629183]
  We give a generic conformal prediction method for sequential prediction. It achieves target empirical coverage guarantees against adversarially chosen data. It is computationally lightweight -- comparable to split conformal prediction.
  arXiv  Detail & Related papers  (2022-06-02T14:33:00Z)
• Don't Explain Noise: Robust Counterfactuals for Randomized Ensembles [50.81061839052459]
  We formalize the generation of robust counterfactual explanations as a probabilistic problem. We show the link between the robustness of ensemble models and the robustness of base learners. Our method achieves high robustness with only a small increase in the distance from counterfactual explanations to their initial observations.
  arXiv  Detail & Related papers  (2022-05-27T17:28:54Z)
• Reinforced Causal Explainer for Graph Neural Networks [112.57265240212001]
  Explainability is crucial for probing graph neural networks (GNNs) We propose a reinforcement learning agent, Reinforced Causal Explainer (RC-Explainer) RC-Explainer generates faithful and concise explanations, and has a better power to unseen graphs.
  arXiv  Detail & Related papers  (2022-04-23T09:13:25Z)
• Provable concept learning for interpretable predictions using variational inference [7.0349768355860895]
  In safety critical applications, practitioners are reluctant to trust neural networks when no interpretable explanations are available. We propose a probabilistic modeling framework to derive (C)oncept (L)earning and (P)rediction (CLAP) We prove that our method is able to identify them while attaining optimal classification accuracy.
  arXiv  Detail & Related papers  (2022-04-01T14:51:38Z)
• Conformal Prediction Sets with Limited False Positives [43.596058175459746]
  We develop a new approach to multi-label conformal prediction in which we aim to output a precise set of promising prediction candidates with a bounded number of incorrect answers. We demonstrate the effectiveness of this approach across a number of classification tasks in natural language processing, computer vision, and computational chemistry.
  arXiv  Detail & Related papers  (2022-02-15T18:52:33Z)
• Deep Probabilistic Graph Matching [72.6690550634166]
  We propose a deep learning-based graph matching framework that works for the original QAP without compromising on the matching constraints. The proposed method is evaluated on three popularly tested benchmarks (Pascal VOC, Willow Object and SPair-71k) and it outperforms all previous state-of-the-arts on all benchmarks.
  arXiv  Detail & Related papers  (2022-01-05T13:37:27Z)
• Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs [89.51365993393787]
  We present BetaE, a probabilistic embedding framework for answering arbitrary FOL queries over a knowledge graph (KG) BetaE is the first method that can handle a complete set of first-order logical operations. We demonstrate the performance of BetaE on answering arbitrary FOL queries on three large, incomplete KGs.
  arXiv  Detail & Related papers  (2020-10-22T06:11:39Z)
• Efficient Conformal Prediction via Cascaded Inference with Expanded Admission [43.596058175459746]
  We present a novel approach for conformal prediction (CP) We aim to identify a set of promising prediction candidates -- in place of a single prediction. This set is guaranteed to contain a correct answer with high probability.
  arXiv  Detail & Related papers  (2020-07-06T23:13:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.