Unified Interpretation of Smoothing Methods for Negative Sampling Loss Functions in Knowledge Graph Embedding

• URL: http://arxiv.org/abs/2407.04251v1
• Date: Fri, 5 Jul 2024 04:38:17 GMT
• Title: Unified Interpretation of Smoothing Methods for Negative Sampling Loss Functions in Knowledge Graph Embedding
• Authors: Xincan Feng, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe,
• Abstract summary: This paper provides theoretical interpretations of the smoothing methods for the Negative Sampling (NS) loss in Knowledge Graphs (KGs) It induces a new NS loss, Triplet Adaptive Negative Sampling (TANS), that can cover the characteristics of the conventional smoothing methods.
• Score: 31.26112477399022
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Knowledge Graphs (KGs) are fundamental resources in knowledge-intensive tasks in NLP. Due to the limitation of manually creating KGs, KG Completion (KGC) has an important role in automatically completing KGs by scoring their links with KG Embedding (KGE). To handle many entities in training, KGE relies on Negative Sampling (NS) loss that can reduce the computational cost by sampling. Since the appearance frequencies for each link are at most one in KGs, sparsity is an essential and inevitable problem. The NS loss is no exception. As a solution, the NS loss in KGE relies on smoothing methods like Self-Adversarial Negative Sampling (SANS) and subsampling. However, it is uncertain what kind of smoothing method is suitable for this purpose due to the lack of theoretical understanding. This paper provides theoretical interpretations of the smoothing methods for the NS loss in KGE and induces a new NS loss, Triplet Adaptive Negative Sampling (TANS), that can cover the characteristics of the conventional smoothing methods. Experimental results of TransE, DistMult, ComplEx, RotatE, HAKE, and HousE on FB15k-237, WN18RR, and YAGO3-10 datasets and their sparser subsets show the soundness of our interpretation and performance improvement by our TANS.

Related papers

• Modality-Aware Negative Sampling for Multi-modal Knowledge Graph Embedding [12.513266782679754]
  Negative sampling (NS) is widely used in knowledge graph embedding (KGE), which aims to generate negative triples to make a positive-negative contrast during training. Existing NS methods are unsuitable when multi-modal information is considered in KGE models. We propose Modality-Aware Negative Sampling (MANS) for multi-modal knowledge graph embedding (MMKGE) to address the mentioned problems.
  arXiv  Detail & Related papers  (2023-04-23T11:22:17Z)
• Normalizing Flow-based Neural Process for Few-Shot Knowledge Graph Completion [69.55700751102376]
  Few-shot knowledge graph completion (FKGC) aims to predict missing facts for unseen relations with few-shot associated facts. Existing FKGC methods are based on metric learning or meta-learning, which often suffer from the out-of-distribution and overfitting problems. In this paper, we propose a normalizing flow-based neural process for few-shot knowledge graph completion (NP-FKGC)
  arXiv  Detail & Related papers  (2023-04-17T11:42:28Z)
• Joint Edge-Model Sparse Learning is Provably Efficient for Graph Neural Networks [89.28881869440433]
  This paper provides the first theoretical characterization of joint edge-model sparse learning for graph neural networks (GNNs) It proves analytically that both sampling important nodes and pruning neurons with the lowest-magnitude can reduce the sample complexity and improve convergence without compromising the test accuracy.
  arXiv  Detail & Related papers  (2023-02-06T16:54:20Z)
• KG-NSF: Knowledge Graph Completion with a Negative-Sample-Free Approach [4.146672630717471]
  We propose bftextKG-NSF, a negative sampling-free framework for learning KG embeddings based on the cross-correlation matrices of embedding vectors. It is shown that the proposed method achieves comparable link prediction performance to negative sampling-based methods while converging much faster.
  arXiv  Detail & Related papers  (2022-07-29T11:39:04Z)
• Explainable Sparse Knowledge Graph Completion via High-order Graph Reasoning Network [111.67744771462873]
  This paper proposes a novel explainable model for sparse Knowledge Graphs (KGs) It combines high-order reasoning into a graph convolutional network, namely HoGRN. It can not only improve the generalization ability to mitigate the information insufficiency issue but also provide interpretability.
  arXiv  Detail & Related papers  (2022-07-14T10:16:56Z)
• Comprehensive Analysis of Negative Sampling in Knowledge Graph Representation Learning [25.664174172917345]
  Negative sampling (NS) loss plays an important role in learning knowledge graph embedding (KGE) to handle a huge number of entities. We theoretically analyzed NS loss to assist hyper parameter tuning and understand the better use of the NS loss in KGE learning. Our empirical analysis on the FB15k-237, WN18RR, and YAGO3-10 datasets showed that the results of actually trained models agree with our theoretical findings.
  arXiv  Detail & Related papers  (2022-06-21T06:51:33Z)
• MEKER: Memory Efficient Knowledge Embedding Representation for Link Prediction and Question Answering [65.62309538202771]
  Knowledge Graphs (KGs) are symbolically structured storages of facts. KG embedding contains concise data used in NLP tasks requiring implicit information about the real world. We propose a memory-efficient KG embedding model, which yields SOTA-comparable performance on link prediction tasks and KG-based Question Answering.
  arXiv  Detail & Related papers  (2022-04-22T10:47:03Z)
• DSKReG: Differentiable Sampling on Knowledge Graph for Recommendation with Relational GNN [59.160401038969795]
  We propose differentiable sampling on Knowledge Graph for Recommendation with GNN (DSKReG) We devise a differentiable sampling strategy, which enables the selection of relevant items to be jointly optimized with the model training procedure. The experimental results demonstrate that our model outperforms state-of-the-art KG-based recommender systems.
  arXiv  Detail & Related papers  (2021-08-26T16:19:59Z)
• Efficient Non-Sampling Knowledge Graph Embedding [21.074002550338296]
  We propose a new framework for KG embedding -- Efficient Non-Sampling Knowledge Graph Embedding (NS-KGE) The basic idea is to consider all of the negative instances in the KG for model learning, and thus to avoid negative sampling. Experiments on benchmark datasets show that our NS-KGE framework can achieve a better performance on efficiency and accuracy over traditional negative sampling based models.
  arXiv  Detail & Related papers  (2021-04-21T23:36:39Z)
• Efficient, Simple and Automated Negative Sampling for Knowledge Graph Embedding [40.97648142355799]
  Negative sampling, which samples negative triplets from non-observed ones in knowledge graph (KG), is an essential step in KG embedding. In this paper, motivated by the observation that negative triplets with large gradients are important but rare, we propose to directly keep track of them with the cache. Our method acts as a "distilled" version of previous GAN-based methods, which does not waste training time on additional parameters to fit the full distribution of negative triplets.
  arXiv  Detail & Related papers  (2020-10-24T14:16:35Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.