GMLM: Bridging Graph Neural Networks and Language Models for Heterophilic Node Classification

• URL: http://arxiv.org/abs/2503.05763v3
• Date: Mon, 02 Jun 2025 08:42:48 GMT
• Title: GMLM: Bridging Graph Neural Networks and Language Models for Heterophilic Node Classification
• Authors: Aarush Sinha, OM Kumar CU,
• Abstract summary: We propose textbfGraph Masked Language Model (GMLM), a novel architecture efficiently combining Graph Neural Networks (GNNs) with Pre-trained Language Models (PLMs)<n>GMLM introduces three key innovations: (i) a textbfdynamic active node selection strategy for scalable PLM text processing; (ii) a GNN-specific textbfcontrastive pretraining stage using soft masking with a learnable graph texttt[MASK] token for robust structural representations; and (iii) a textbfdedicated fusion
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Integrating structured graph data with rich textual information from nodes poses a significant challenge, particularly for heterophilic node classification. Current approaches often struggle with computational costs or effective fusion of disparate modalities. We propose \textbf{Graph Masked Language Model (GMLM)}, a novel architecture efficiently combining Graph Neural Networks (GNNs) with Pre-trained Language Models (PLMs). GMLM introduces three key innovations: (i) a \textbf{dynamic active node selection} strategy for scalable PLM text processing; (ii) a GNN-specific \textbf{contrastive pretraining stage} using soft masking with a learnable graph \texttt{[MASK]} token for robust structural representations; and (iii) a \textbf{dedicated fusion module} integrating RGCN-based GNN embeddings with PLM (GTE-Small \& DistilBERT) embeddings. Extensive experiments on heterophilic benchmarks (Cornell, Wisconsin, Texas) demonstrate GMLM's superiority. Notably, GMLM(DistilBERT) achieves significant performance gains, improving accuracy by over \textbf{4.7\%} on Cornell and over \textbf{2.0\%} on Texas compared to the previous best-performing baselines. This work underscores the benefits of targeted PLM engagement and modality-specific pretraining for improved, efficient learning on text-rich graphs.

Related papers

• Integrating Structural and Semantic Signals in Text-Attributed Graphs with BiGTex [0.16385815610837165]
  BiGTex is a novel architecture that tightly integrates GNNs and LLMs through stacked Graph-Text Fusion Units. BiGTex achieves state-of-the-art performance in node classification and generalizes effectively to link prediction.
  arXiv  Detail & Related papers  (2025-04-16T20:25:11Z)
• LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Models [54.82915844507371]
  Text-Attributed Graphs (TAGs) are ubiquitous in real-world scenarios. Despite large efforts to integrate Large Language Models (LLMs) and Graph Neural Networks (GNNs) for TAGs, existing approaches suffer from decoupled architectures. We propose PromptGFM, a versatile GFM for TAGs grounded in graph vocabulary learning.
  arXiv  Detail & Related papers  (2025-03-05T09:45:22Z)
• Deep Semantic Graph Learning via LLM based Node Enhancement [5.312946761836463]
  Large Language Models (LLMs) have demonstrated superior capabilities in understanding text semantics.<n>This paper proposes a novel framework that combines Graph Transformer architecture with LLM-enhanced node features.
  arXiv  Detail & Related papers  (2025-02-11T21:55:46Z)
• Can Graph Neural Networks Learn Language with Extremely Weak Text Supervision? [62.12375949429938]
  Building transferable Graph Neural Networks (GNNs) with CLIP pipeline is challenging because of three fundamental issues. We leverage multi-modal prompt learning to effectively adapt pre-trained GNN to downstream tasks and data. Our new paradigm embeds the graphs directly in the same space as the Large Language Models (LLMs) by learning both graph prompts and text prompts simultaneously.
  arXiv  Detail & Related papers  (2024-12-11T08:03:35Z)
• GL-Fusion: Rethinking the Combination of Graph Neural Network and Large Language model [63.774726052837266]
  We introduce a new architecture that deeply integrates Graph Neural Networks (GNNs) with Large Language Models (LLMs)<n>We introduce three key innovations: (1) Structure-Aware Transformers, which incorporate GNN's message-passing capabilities directly into LLM's transformer layers; (2) Graph-Text Cross-Attention, which processes full, uncompressed text from graph nodes and edges; and (3) GNN-LLM Twin Predictor, enabling LLM's flexible autoregressive generation alongside GNN's scalable one-pass prediction.
  arXiv  Detail & Related papers  (2024-12-08T05:49:58Z)
• Bridging Large Language Models and Graph Structure Learning Models for Robust Representation Learning [22.993015048941444]
  Graph representation learning is crucial for real-world applications but often encounters pervasive noise. We introduce LangGSL, a framework that integrates the complementary strengths of pre-trained language models and graph structure learning models.
  arXiv  Detail & Related papers  (2024-10-15T22:43:32Z)
• How to Make LLMs Strong Node Classifiers? [70.14063765424012]
  Language Models (LMs) are challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs)<n>We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art (SOTA) GNNs on node classification tasks.
  arXiv  Detail & Related papers  (2024-10-03T08:27:54Z)
• All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networks [51.19110891434727]
  Large Language Models (LLMs) with pretrained knowledge and powerful semantic comprehension abilities have recently shown a remarkable ability to benefit applications using vision and text data. E-LLaGNN is a framework with an on-demand LLM service that enriches message passing procedure of graph learning by enhancing a limited fraction of nodes from the graph.
  arXiv  Detail & Related papers  (2024-07-20T22:09:42Z)
• A Pure Transformer Pretraining Framework on Text-attributed Graphs [50.833130854272774]
  We introduce a feature-centric pretraining perspective by treating graph structure as a prior. Our framework, Graph Sequence Pretraining with Transformer (GSPT), samples node contexts through random walks. GSPT can be easily adapted to both node classification and link prediction, demonstrating promising empirical success on various datasets.
  arXiv  Detail & Related papers  (2024-06-19T22:30:08Z)
• Multi-View Empowered Structural Graph Wordification for Language Models [12.22063024099311]
  We introduce an end-to-end modality-aligning framework for LLM-graph alignment: Dual-Residual Vector Quantized-Variational AutoEncoder, namely Dr.E.<n>Our approach is purposefully designed to facilitate token-level alignment with LLMs, enabling an effective translation of the intrinsic'of graphs into comprehensible natural language.<n>Our framework ensures certain visual interpretability, efficiency, and robustness, marking the promising successful endeavor to achieve token-level alignment between LLMs and GNNs.
  arXiv  Detail & Related papers  (2024-06-19T16:43:56Z)
• Parameter-Efficient Tuning Large Language Models for Graph Representation Learning [62.26278815157628]
  We introduce Graph-aware. Efficient Fine-Tuning - GPEFT, a novel approach for efficient graph representation learning. We use a graph neural network (GNN) to encode structural information from neighboring nodes into a graph prompt. We validate our approach through comprehensive experiments conducted on 8 different text-rich graphs, observing an average improvement of 2% in hit@1 and Mean Reciprocal Rank (MRR) in link prediction evaluations.
  arXiv  Detail & Related papers  (2024-04-28T18:36:59Z)
• GraphEdit: Large Language Models for Graph Structure Learning [14.16155596597421]
  Graph Structure Learning (GSL) focuses on capturing intrinsic dependencies and interactions among nodes in graph-structured data.<n>Existing GSL methods heavily depend on explicit graph structural information as supervision signals.<n>We propose GraphEdit, an approach that leverages large language models (LLMs) to learn complex node relationships in graph-structured data.
  arXiv  Detail & Related papers  (2024-02-23T08:29:42Z)
• Graph Language Models [18.75364157933661]
  We introduce a novel LM type, the Graph Language Model (GLM), that integrates the strengths of both approaches and mitigates their weaknesses. We design the GLM's architecture to incorporate graph biases, thereby promoting effective knowledge distribution within the graph. Empirical evaluations on relation classification tasks show that GLM embeddings surpass both LM- and GNN-based baselines in supervised and zero-shot setting.
  arXiv  Detail & Related papers  (2024-01-13T16:09:49Z)
• Efficient and effective training of language and graph neural network models [36.00479096375565]
  We put forth an efficient and effective framework termed language model GNN (LM-GNN) to jointly train large-scale language models and graph neural networks. The effectiveness in our framework is achieved by applying stage-wise fine-tuning of the BERT model first with heterogenous graph information and then with a GNN model. We evaluate the LM-GNN framework in different datasets performance and showcase the effectiveness of the proposed approach.
  arXiv  Detail & Related papers  (2022-06-22T00:23:37Z)
• A Robust Stacking Framework for Training Deep Graph Models with Multifaceted Node Features [61.92791503017341]
  Graph Neural Networks (GNNs) with numerical node features and graph structure as inputs have demonstrated superior performance on various supervised learning tasks with graph data. The best models for such data types in most standard supervised learning settings with IID (non-graph) data are not easily incorporated into a GNN. Here we propose a robust stacking framework that fuses graph-aware propagation with arbitrary models intended for IID data.
  arXiv  Detail & Related papers  (2022-06-16T22:46:33Z)
• MGAE: Masked Autoencoders for Self-Supervised Learning on Graphs [55.66953093401889]
  Masked graph autoencoder (MGAE) framework to perform effective learning on graph structure data. Taking insights from self-supervised learning, we randomly mask a large proportion of edges and try to reconstruct these missing edges during training.
  arXiv  Detail & Related papers  (2022-01-07T16:48:07Z)
• GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graph [53.70520466556453]
  We propose GraphFormers, where layerwise GNN components are nested alongside the transformer blocks of language models. With the proposed architecture, the text encoding and the graph aggregation are fused into an iterative workflow. In addition, a progressive learning strategy is introduced, where the model is successively trained on manipulated data and original data to reinforce its capability of integrating information on graph.
  arXiv  Detail & Related papers  (2021-05-06T12:20:41Z)
• Exploiting Structured Knowledge in Text via Graph-Guided Representation Learning [73.0598186896953]
  We present two self-supervised tasks learning over raw text with the guidance from knowledge graphs. Building upon entity-level masked language models, our first contribution is an entity masking scheme. In contrast to existing paradigms, our approach uses knowledge graphs implicitly, only during pre-training.
  arXiv  Detail & Related papers  (2020-04-29T14:22:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.