Masked Language Models are Good Heterogeneous Graph Generalizers

• URL: http://arxiv.org/abs/2506.06157v2
• Date: Wed, 30 Jul 2025 03:58:06 GMT
• Title: Masked Language Models are Good Heterogeneous Graph Generalizers
• Authors: Jinyu Yang, Cheng Yang, Shanyuan Cui, Zeyuan Guo, Liangwei Yang, Muhan Zhang, Zhiqiang Zhang, Chuan Shi,
• Abstract summary: Masked Language Modeling-based method called LLM4HG.<n>Uses metapath-based sequences instead of HG tokens to extract structural and semantic information.
• Score: 54.08788279971086
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Heterogeneous graph neural networks (HGNNs) excel at capturing structural and semantic information in heterogeneous graphs (HGs), while struggling to generalize across domains and tasks. With the rapid advancement of large language models (LLMs), a recent study explored the integration of HGNNs with LLMs for generalizable heterogeneous graph learning. However, this approach typically encodes structural information as HG tokens using HGNNs, and disparities in embedding spaces between HGNNs and LLMs have been shown to bias the LLM's comprehension of HGs. Moreover, since these HG tokens are often derived from node-level tasks, the model's ability to generalize across tasks remains limited. To this end, we propose a simple yet effective Masked Language Modeling-based method, called MLM4HG. MLM4HG introduces metapath-based textual sequences instead of HG tokens to extract structural and semantic information inherent in HGs, and designs customized textual templates to unify different graph tasks into a coherent cloze-style 'mask' token prediction paradigm. Specifically,MLM4HG first converts HGs from various domains to texts based on metapaths, and subsequently combines them with the unified task texts to form a HG-based corpus. Moreover, the corpus is fed into a pretrained LM for fine-tuning with a constrained target vocabulary, enabling the fine-tuned LM to generalize to unseen target HGs. Extensive cross-domain and multi-task experiments on four real-world datasets demonstrate the superior generalization performance of MLM4HG over state-of-the-art methods in both few-shot and zero-shot scenarios. Our code is available at https://github.com/BUPT-GAMMA/MLM4HG.

Related papers

• LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Models [54.82915844507371]
  Text-Attributed Graphs (TAGs) are ubiquitous in real-world scenarios.<n>Despite large efforts to integrate Large Language Models (LLMs) and Graph Neural Networks (GNNs) for TAGs, existing approaches suffer from decoupled architectures.<n>We propose PromptGFM, a versatile GFM for TAGs grounded in graph vocabulary learning.
  arXiv  Detail & Related papers  (2025-03-05T09:45:22Z)
• GMLM: Bridging Graph Neural Networks and Language Models for Heterophilic Node Classification [0.0]
  We propose a framework designed for the efficient and effective fusion of graph structure and text semantics.<n> Experiments on five heterophilic benchmarks show GMLM achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2025-02-24T07:44:01Z)
• Model Generalization on Text Attribute Graphs: Principles with Large Language Models [14.657522068231138]
  Large language models (LLMs) have been introduced to graph learning, aiming to extend their zero-shot generalization success to tasks where labeled graph data is scarce.<n>We develop a framework for inference over text-attributed graphs (TAGs) based on task-adaptive embeddings and a generalizable graph information aggregation mechanism.<n> Evaluations on 11 real-world TAG benchmarks demonstrate that LLM-BP significantly outperforms existing approaches.
  arXiv  Detail & Related papers  (2025-02-17T14:31:00Z)
• Deep Semantic Graph Learning via LLM based Node Enhancement [5.312946761836463]
  Large Language Models (LLMs) have demonstrated superior capabilities in understanding text semantics.<n>This paper proposes a novel framework that combines Graph Transformer architecture with LLM-enhanced node features.
  arXiv  Detail & Related papers  (2025-02-11T21:55:46Z)
• GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model [63.774726052837266]
  We introduce a new architecture that deeply integrates Graph Neural Networks (GNNs) with Large Language Models (LLMs)<n>We introduce three key innovations: (1) Structure-Aware Transformers, which incorporate GNN's message-passing capabilities directly into LLM's transformer layers; (2) Graph-Text Cross-Attention, which processes full, uncompressed text from graph nodes and edges; and (3) GNN-LLM Twin Predictor, enabling LLM's flexible autoregressive generation alongside GNN's scalable one-pass prediction.
  arXiv  Detail & Related papers  (2024-12-08T05:49:58Z)
• How to Make LLMs Strong Node Classifiers? [70.14063765424012]
  Language Models (LMs) are challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs)<n>We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art (SOTA) GNNs on node classification tasks.
  arXiv  Detail & Related papers  (2024-10-03T08:27:54Z)
• GSVA: Generalized Segmentation via Multimodal Large Language Models [72.57095903188922]
  Generalized Referring Expression (GRES) extends the scope of classic RES to refer to multiple objects in one expression or identify the empty targets absent in the image. Current solutions to GRES remain unsatisfactory since segmentation MLLMs cannot correctly handle the cases where users might reference multiple subjects in a singular prompt. We propose Generalized Vision Assistant (GSVA) to address this gap.
  arXiv  Detail & Related papers  (2023-12-15T02:54:31Z)
• Cross-heterogeneity Graph Few-shot Learning [9.80898395055038]
  We propose a novel model for Cross-heterogeneity Graph Few-shot Learning, namely CGFL. In CGFL, we first extract meta-patterns to capture heterogeneous information. Then, we propose a score module to measure the informativeness of labeled samples and determine the transferability of each source HG.
  arXiv  Detail & Related papers  (2023-08-10T01:25:28Z)
• Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning [51.90524745663737]
  A key innovation is our use of explanations as features, which can be used to boost GNN performance on downstream tasks. Our method achieves state-of-the-art results on well-established TAG datasets. Our method significantly speeds up training, achieving a 2.88 times improvement over the closest baseline on ogbn-arxiv.
  arXiv  Detail & Related papers  (2023-05-31T03:18:03Z)
• Hierarchical Memory Learning for Fine-Grained Scene Graph Generation [49.39355372599507]
  This paper proposes a novel Hierarchical Memory Learning (HML) framework to learn the model from simple to complex. After the autonomous partition of coarse and fine predicates, the model is first trained on the coarse predicates and then learns the fine predicates.
  arXiv  Detail & Related papers  (2022-03-14T08:01:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.