Sem@$K$: Is my knowledge graph embedding model semantic-aware?

• URL: http://arxiv.org/abs/2301.05601v2
• Date: Thu, 7 Dec 2023 16:13:24 GMT
• Title: Sem@$K$: Is my knowledge graph embedding model semantic-aware?
• Authors: Nicolas Hubert, Pierre Monnin, Armelle Brun, Davy Monticolo
• Abstract summary: We extend our previously introduced metric Sem@K that measures the capability of models to predict valid entities w.r.t. domain and range constraints. Our experiments show that Sem@K provides a new perspective on KGEM quality. Some KGEMs are inherently better than others, but this semantic superiority is not indicative of their performance w.r.t. rank-based metrics.
• Score: 1.8024397171920883
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Using knowledge graph embedding models (KGEMs) is a popular approach for predicting links in knowledge graphs (KGs). Traditionally, the performance of KGEMs for link prediction is assessed using rank-based metrics, which evaluate their ability to give high scores to ground-truth entities. However, the literature claims that the KGEM evaluation procedure would benefit from adding supplementary dimensions to assess. That is why, in this paper, we extend our previously introduced metric Sem@K that measures the capability of models to predict valid entities w.r.t. domain and range constraints. In particular, we consider a broad range of KGs and take their respective characteristics into account to propose different versions of Sem@K. We also perform an extensive study to qualify the abilities of KGEMs as measured by our metric. Our experiments show that Sem@K provides a new perspective on KGEM quality. Its joint analysis with rank-based metrics offers different conclusions on the predictive power of models. Regarding Sem@K, some KGEMs are inherently better than others, but this semantic superiority is not indicative of their performance w.r.t. rank-based metrics. In this work, we generalize conclusions about the relative performance of KGEMs w.r.t. rank-based and semantic-oriented metrics at the level of families of models. The joint analysis of the aforementioned metrics gives more insight into the peculiarities of each model. This work paves the way for a more comprehensive evaluation of KGEM adequacy for specific downstream tasks.

Related papers

• Guardians of the Machine Translation Meta-Evaluation: Sentinel Metrics Fall In! [80.3129093617928]
  Annually, at the Conference of Machine Translation (WMT), the Metrics Shared Task organizers conduct the meta-evaluation of Machine Translation (MT) metrics. This work highlights two issues with the meta-evaluation framework currently employed in WMT, and assesses their impact on the metrics rankings. We introduce the concept of sentinel metrics, which are designed explicitly to scrutinize the meta-evaluation process's accuracy, robustness, and fairness.
  arXiv  Detail & Related papers  (2024-08-25T13:29:34Z)
• Benchmarks as Microscopes: A Call for Model Metrology [76.64402390208576]
  Modern language models (LMs) pose a new challenge in capability assessment. To be confident in our metrics, we need a new discipline of model metrology.
  arXiv  Detail & Related papers  (2024-07-22T17:52:12Z)
• Do Similar Entities have Similar Embeddings? [2.9498907601878974]
  Knowledge graph embedding models (KGEMs) developed for link prediction learn vector representations for entities in a knowledge graph, known as embeddings. A common assumption is the KGE entity similarity assumption, which states that these KGEMs retain the graph's structure within their embedding space. Yet, the relation of entity similarity and similarity in the embedding space has rarely been formally evaluated. This paper challenges the prevailing assumption that entity similarity in the graph is inherently mirrored in the embedding space.
  arXiv  Detail & Related papers  (2023-12-16T08:08:36Z)
• Rethinking Language Models as Symbolic Knowledge Graphs [7.192286645674803]
  Symbolic knowledge graphs (KGs) play a pivotal role in knowledge-centric applications such as search, question answering and recommendation. We construct nine qualitative benchmarks that encompass a spectrum of attributes including symmetry, asymmetry, hierarchy, bidirectionality, compositionality, paths, entity-centricity, bias and ambiguity.
  arXiv  Detail & Related papers  (2023-08-25T21:25:08Z)
• Schema First! Learn Versatile Knowledge Graph Embeddings by Capturing Semantics with MASCHInE [3.174882428337821]
  Knowledge graph embedding models (KGEMs) have gained considerable traction in recent years. In this work, we design protographs -- small, modified versions of a KG that leverage RDF/S information. The learnt protograph-based embeddings are meant to encapsulate the semantics of a KG, and can be leveraged in learning KGEs that, in turn, also better capture semantics.
  arXiv  Detail & Related papers  (2023-06-06T13:22:54Z)
• Joint Metrics Matter: A Better Standard for Trajectory Forecasting [67.1375677218281]
  Multi-modal trajectory forecasting methods evaluate using single-agent metrics (marginal metrics) Only focusing on marginal metrics can lead to unnatural predictions, such as colliding trajectories or diverging trajectories for people who are clearly walking together as a group. We present the first comprehensive evaluation of state-of-the-art trajectory forecasting methods with respect to multi-agent metrics (joint metrics): JADE, JFDE, and collision rate.
  arXiv  Detail & Related papers  (2023-05-10T16:27:55Z)
• KGxBoard: Explainable and Interactive Leaderboard for Evaluation of Knowledge Graph Completion Models [76.01814380927507]
  KGxBoard is an interactive framework for performing fine-grained evaluation on meaningful subsets of the data. In our experiments, we highlight the findings with the use of KGxBoard, which would have been impossible to detect with standard averaged single-score metrics.
  arXiv  Detail & Related papers  (2022-08-23T15:11:45Z)
• Knowledge Graph Embedding Methods for Entity Alignment: An Experimental Review [7.241438112282638]
  We conduct the first meta-level analysis of popular embedding methods for entity alignment. Our analysis reveals statistically significant correlations of different embedding methods with various meta-features extracted by KGs. We rank them in a statistically significant way according to their effectiveness across all real-world KGs of our testbed.
  arXiv  Detail & Related papers  (2022-03-17T12:11:58Z)
• Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary [65.37544133256499]
  We propose a metric to evaluate the content quality of a summary using question-answering (QA) We demonstrate the experimental benefits of QA-based metrics through an analysis of our proposed metric, QAEval.
  arXiv  Detail & Related papers  (2020-10-01T15:33:09Z)
• Evaluation Metrics for Conditional Image Generation [100.69766435176557]
  We present two new metrics for evaluating generative models in the class-conditional image generation setting. A theoretical analysis shows the motivation behind each proposed metric and links the novel metrics to their unconditional counterparts. We provide an extensive empirical evaluation, comparing the metrics to their unconditional variants and to other metrics, and utilize them to analyze existing generative models.
  arXiv  Detail & Related papers  (2020-04-26T12:15:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.