KG-BiLM: Knowledge Graph Embedding via Bidirectional Language Models

• URL: http://arxiv.org/abs/2506.03576v1
• Date: Wed, 04 Jun 2025 04:47:24 GMT
• Title: KG-BiLM: Knowledge Graph Embedding via Bidirectional Language Models
• Authors: Zirui Chen, Xin Wang, Zhao Li, Wenbin Guo, Dongxiao He,
• Abstract summary: KG-BiLM is a bidirectional LM framework that fuses structural cues from KGs with the semantic expressiveness of generative transformers.<n> KG-BiLM incorporates three key components: (i) Bidirectional Knowledge Attention, which removes the causal mask to enable full interaction among all tokens and entities; (ii) Knowledge-Masked Prediction, which encourages the model to leverage both local semantic contexts and global graph connectivity; and (iii) Contrastive Graph Semantic Aggregation, which preserves KG structure via contrastive alignment of sampled sub-graph representations.
• Score: 16.2387767061355
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Recent advances in knowledge representation learning (KRL) highlight the urgent necessity to unify symbolic knowledge graphs (KGs) with language models (LMs) for richer semantic understanding. However, existing approaches typically prioritize either graph structure or textual semantics, leaving a gap: a unified framework that simultaneously captures global KG connectivity, nuanced linguistic context, and discriminative reasoning semantics. To bridge this gap, we introduce KG-BiLM, a bidirectional LM framework that fuses structural cues from KGs with the semantic expressiveness of generative transformers. KG-BiLM incorporates three key components: (i) Bidirectional Knowledge Attention, which removes the causal mask to enable full interaction among all tokens and entities; (ii) Knowledge-Masked Prediction, which encourages the model to leverage both local semantic contexts and global graph connectivity; and (iii) Contrastive Graph Semantic Aggregation, which preserves KG structure via contrastive alignment of sampled sub-graph representations. Extensive experiments on standard benchmarks demonstrate that KG-BiLM outperforms strong baselines in link prediction, especially on large-scale graphs with complex multi-hop relations - validating its effectiveness in unifying structural information and textual semantics.

Related papers

• SEMMA: A Semantic Aware Knowledge Graph Foundation Model [34.001718474850676]
  Knowledge Graph Foundation Models (KGFMs) have shown promise in enabling zero-shot reasoning over unseen graphs by learning transferable patterns.<n>We introduce SEMMA, a dual-module KGFM that integrates transferable textual semantics alongside structure.<n>We show that in more challenging generalization settings, where the test-time relation vocabulary is entirely unseen, structural methods collapse while SEMMA is 2x more effective.
  arXiv  Detail & Related papers  (2025-05-26T18:15:25Z)
• GLTW: Joint Improved Graph Transformer and LLM via Three-Word Language for Knowledge Graph Completion [52.026016846945424]
  We propose a new method called GLTW, which encodes the structural information of KGs and merges it with Large Language Models.<n>Specifically, we introduce an improved Graph Transformer (iGT) that effectively encodes subgraphs with both local and global structural information.<n>Also, we develop a subgraph-based multi-classification training objective, using all entities within KG as classification objects, to boost learning efficiency.
  arXiv  Detail & Related papers  (2025-02-17T06:02:59Z)
• Self-supervised Quantized Representation for Seamlessly Integrating Knowledge Graphs with Large Language Models [17.88134311726175]
  We propose a framework to learn and apply quantized codes for each entity, aiming for the seamless integration of Knowledge Graphs with Large Language Models.<n>Experiment results demonstrate that SSQR outperforms existing unsupervised quantized methods, producing more distinguishable codes.<n>The fine-tuned LLaMA2 and LLaMA3.1 also have superior performance on KG link prediction and triple classification tasks.
  arXiv  Detail & Related papers  (2025-01-30T03:40:20Z)
• Bridge: A Unified Framework to Knowledge Graph Completion via Language Models and Knowledge Representation [14.801411392475439]
  We propose a novel framework called Bridge, which jointly encodes structural and semantic information of Knowledge Graphs (KGs)<n>Specifically, we strategically encode entities and relations separately by PLMs to better utilize the semantic knowledge of PLMs.<n>To bridge the gap between KGs and PLMs, we employ a self-supervised representation learning method called BYOL to fine-tune PLMs with two different views of a triple.
  arXiv  Detail & Related papers  (2024-11-11T01:59:04Z)
• Bridging Local Details and Global Context in Text-Attributed Graphs [62.522550655068336]
  GraphBridge is a framework that bridges local and global perspectives by leveraging contextual textual information. Our method achieves state-of-theart performance, while our graph-aware token reduction module significantly enhances efficiency and solves scalability issues.
  arXiv  Detail & Related papers  (2024-06-18T13:35:25Z)
• Knowledge Graph Large Language Model (KG-LLM) for Link Prediction [43.55117421485917]
  We introduce the Knowledge Graph Large Language Model (KG-LLM), a novel framework that leverages large language models (LLMs) for knowledge graph tasks. We first convert structured knowledge graph data into natural language and then use these natural language prompts to fine-tune LLMs. To show the efficacy of the KG-LLM Framework, we fine-tune three leading LLMs within this framework, including Flan-T5, LLaMa2 and Gemma.
  arXiv  Detail & Related papers  (2024-03-12T04:47:29Z)
• Contextualization Distillation from Large Language Model for Knowledge Graph Completion [51.126166442122546]
  We introduce the Contextualization Distillation strategy, a plug-in-and-play approach compatible with both discriminative and generative KGC frameworks. Our method begins by instructing large language models to transform compact, structural triplets into context-rich segments. Comprehensive evaluations across diverse datasets and KGC techniques highlight the efficacy and adaptability of our approach.
  arXiv  Detail & Related papers  (2024-01-28T08:56:49Z)
• GreaseLM: Graph REASoning Enhanced Language Models for Question Answering [159.9645181522436]
  GreaseLM is a new model that fuses encoded representations from pretrained LMs and graph neural networks over multiple layers of modality interaction operations. We show that GreaseLM can more reliably answer questions that require reasoning over both situational constraints and structured knowledge, even outperforming models 8x larger.
  arXiv  Detail & Related papers  (2022-01-21T19:00:05Z)
• Knowledge Graph Augmented Network Towards Multiview Representation Learning for Aspect-based Sentiment Analysis [96.53859361560505]
  We propose a knowledge graph augmented network (KGAN) to incorporate external knowledge with explicitly syntactic and contextual information. KGAN captures the sentiment feature representations from multiple perspectives, i.e., context-, syntax- and knowledge-based. Experiments on three popular ABSA benchmarks demonstrate the effectiveness and robustness of our KGAN.
  arXiv  Detail & Related papers  (2022-01-13T08:25:53Z)
• KELM: Knowledge Enhanced Pre-Trained Language Representations with Message Passing on Hierarchical Relational Graphs [26.557447199727758]
  We propose a novel knowledge-aware language model framework based on fine-tuning process. Our model can efficiently incorporate world knowledge from KGs into existing language models such as BERT.
  arXiv  Detail & Related papers  (2021-09-09T12:39:17Z)
• On the Role of Conceptualization in Commonsense Knowledge Graph Construction [59.39512925793171]
  Commonsense knowledge graphs (CKGs) like Atomic and ASER are substantially different from conventional KGs. We introduce to CKG construction methods conceptualization to view entities mentioned in text as instances of specific concepts or vice versa. Our methods can effectively identify plausible triples and expand the KG by triples of both new nodes and edges of high diversity and novelty.
  arXiv  Detail & Related papers  (2020-03-06T14:35:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.