Efficient Non-Sampling Knowledge Graph Embedding

• URL: http://arxiv.org/abs/2104.10796v1
• Date: Wed, 21 Apr 2021 23:36:39 GMT
• Title: Efficient Non-Sampling Knowledge Graph Embedding
• Authors: Zelong Li, Jianchao Ji, Zuohui Fu, Yingqiang Ge, Shuyuan Xu, Chong Chen, Yongfeng Zhang
• Abstract summary: We propose a new framework for KG embedding -- Efficient Non-Sampling Knowledge Graph Embedding (NS-KGE) The basic idea is to consider all of the negative instances in the KG for model learning, and thus to avoid negative sampling. Experiments on benchmark datasets show that our NS-KGE framework can achieve a better performance on efficiency and accuracy over traditional negative sampling based models.
• Score: 21.074002550338296
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge Graph (KG) is a flexible structure that is able to describe the complex relationship between data entities. Currently, most KG embedding models are trained based on negative sampling, i.e., the model aims to maximize some similarity of the connected entities in the KG, while minimizing the similarity of the sampled disconnected entities. Negative sampling helps to reduce the time complexity of model learning by only considering a subset of negative instances, which may fail to deliver stable model performance due to the uncertainty in the sampling procedure. To avoid such deficiency, we propose a new framework for KG embedding -- Efficient Non-Sampling Knowledge Graph Embedding (NS-KGE). The basic idea is to consider all of the negative instances in the KG for model learning, and thus to avoid negative sampling. The framework can be applied to square-loss based knowledge graph embedding models or models whose loss can be converted to a square loss. A natural side-effect of this non-sampling strategy is the increased computational complexity of model learning. To solve the problem, we leverage mathematical derivations to reduce the complexity of non-sampling loss function, which eventually provides us both better efficiency and better accuracy in KG embedding compared with existing models. Experiments on benchmark datasets show that our NS-KGE framework can achieve a better performance on efficiency and accuracy over traditional negative sampling based models, and that the framework is applicable to a large class of knowledge graph embedding models.

Related papers

• How to Turn Your Knowledge Graph Embeddings into Generative Models [10.466244652188777]
  Some of the most successful knowledge graph embedding (KGE) models for link prediction can be interpreted as energy-based models. This work re-interprets the score functions of these KGEs as circuits. Our interpretation comes with little or no loss of performance for link prediction.
  arXiv  Detail & Related papers  (2023-05-25T11:30:27Z)
• Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
  Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data. We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
  arXiv  Detail & Related papers  (2023-03-06T14:49:59Z)
• LightGCL: Simple Yet Effective Graph Contrastive Learning for Recommendation [9.181689366185038]
  Graph neural clustering network (GNN) is a powerful learning approach for graph-based recommender systems. In this paper, we propose a simple yet effective graph contrastive learning paradigm LightGCL.
  arXiv  Detail & Related papers  (2023-02-16T10:16:21Z)
• GreenKGC: A Lightweight Knowledge Graph Completion Method [32.528770408502396]
  GreenKGC aims to discover missing relationships between entities in knowledge graphs. It consists of three modules: representation learning, feature pruning, and decision learning. In low dimensions, GreenKGC can outperform SOTA methods in most datasets.
  arXiv  Detail & Related papers  (2022-08-19T03:33:45Z)
• Sparse MoEs meet Efficient Ensembles [49.313497379189315]
  We study the interplay of two popular classes of such models: ensembles of neural networks and sparse mixture of experts (sparse MoEs) We present Efficient Ensemble of Experts (E$3$), a scalable and simple ensemble of sparse MoEs that takes the best of both classes of models, while using up to 45% fewer FLOPs than a deep ensemble.
  arXiv  Detail & Related papers  (2021-10-07T11:58:35Z)
• T-LoHo: A Bayesian Regularization Model for Structured Sparsity and Smoothness on Graphs [0.0]
  In graph-structured data, structured sparsity and smoothness tend to cluster together. We propose a new prior for high dimensional parameters with graphical relations. We use it to detect structured sparsity and smoothness simultaneously.
  arXiv  Detail & Related papers  (2021-07-06T10:10:03Z)
• Rethinking InfoNCE: How Many Negative Samples Do You Need? [54.146208195806636]
  We study how many negative samples are optimal for InfoNCE in different scenarios via a semi-quantitative theoretical framework. We estimate the optimal negative sampling ratio using the $K$ value that maximizes the training effectiveness function.
  arXiv  Detail & Related papers  (2021-05-27T08:38:29Z)
• Contextual Dropout: An Efficient Sample-Dependent Dropout Module [60.63525456640462]
  Dropout has been demonstrated as a simple and effective module to regularize the training process of deep neural networks. We propose contextual dropout with an efficient structural design as a simple and scalable sample-dependent dropout module. Our experimental results show that the proposed method outperforms baseline methods in terms of both accuracy and quality of uncertainty estimation.
  arXiv  Detail & Related papers  (2021-03-06T19:30:32Z)
• MixKD: Towards Efficient Distillation of Large-scale Language Models [129.73786264834894]
  We propose MixKD, a data-agnostic distillation framework, to endow the resulting model with stronger generalization ability. We prove from a theoretical perspective that under reasonable conditions MixKD gives rise to a smaller gap between the error and the empirical error. Experiments under a limited-data setting and ablation studies further demonstrate the advantages of the proposed approach.
  arXiv  Detail & Related papers  (2020-11-01T18:47:51Z)
• LowFER: Low-rank Bilinear Pooling for Link Prediction [4.110108749051657]
  We propose a factorized bilinear pooling model, commonly used in multi-modal learning, for better fusion of entities and relations. Our model naturally generalizes decomposition Tucker based TuckER model, which has been shown to generalize other models. We evaluate on real-world datasets, reaching on par or state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-08-25T07:33:52Z)
• Reinforced Negative Sampling over Knowledge Graph for Recommendation [106.07209348727564]
  We develop a new negative sampling model, Knowledge Graph Policy Network (kgPolicy), which works as a reinforcement learning agent to explore high-quality negatives. kgPolicy navigates from the target positive interaction, adaptively receives knowledge-aware negative signals, and ultimately yields a potential negative item to train the recommender.
  arXiv  Detail & Related papers  (2020-03-12T12:44:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.