Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations

• URL: http://arxiv.org/abs/2506.21682v1
• Date: Thu, 26 Jun 2025 18:10:28 GMT
• Title: Do We Really Need GNNs with Explicit Structural Modeling? MLPs Suffice for Language Model Representations
• Authors: Li Zhou, Hao Jiang, Junjie Li, Zefeng Zhao, Feng Jiang, Wenyu Chen, Haizhou Li,
• Abstract summary: Graph Neural Networks (GNNs) fail to fully utilize structural information, whereas Multi-Layer Perceptrons (MLPs) exhibit a surprising ability in structure-aware tasks.<n>This paper introduces a comprehensive probing framework from an information-theoretic perspective.
• Score: 50.45261187796993
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Explicit structural information has been proven to be encoded by Graph Neural Networks (GNNs), serving as auxiliary knowledge to enhance model capabilities and improve performance in downstream NLP tasks. However, recent studies indicate that GNNs fail to fully utilize structural information, whereas Multi-Layer Perceptrons (MLPs), despite lacking the message-passing mechanisms inherent to GNNs, exhibit a surprising ability in structure-aware tasks. Motivated by these findings, this paper introduces a comprehensive probing framework from an information-theoretic perspective. The framework is designed to systematically assess the role of explicit structural modeling in enhancing language model (LM) representations and to investigate the potential of MLPs as efficient and scalable alternatives to GNNs. We extend traditional probing classifiers by incorporating a control module that allows for selective use of either the full GNN model or its decoupled components, specifically, the message-passing and feature-transformation operations.This modular approach isolates and assesses the individual contributions of these operations, avoiding confounding effects from the complete GNN architecture. Using the Edge Probing Suite, a diagnostic tool for evaluating the linguistic knowledge encoded in LMs, we find that MLPs, when used as feature-transformation modules, consistently improve the linguistic knowledge captured in LM representations across different architectures. They effectively encode both syntactic and semantic patterns. Similarly, GNNs that incorporate feature-transformation operations show beneficial effects. In contrast, models that rely solely on message-passing operations tend to underperform, often leading to negative impacts on probing task performance.

Related papers

• Enhancing Knowledge Graph Completion with GNN Distillation and Probabilistic Interaction Modeling [14.550518793230404]
  Knowledge graphs serve as fundamental structures for organizing interconnected data across diverse domains.<n>Existing methods face critical challenges: deep graph neural networks (GNNs) suffer from over-smoothing, while embedding-based models fail to capture abstract relational features.<n>This study proposes a unified framework that integrates GNN distillation and abstract probabilistic interaction modeling (APIM)
  arXiv  Detail & Related papers  (2025-05-18T07:22:53Z)
• LLM Enhancers for GNNs: An Analysis from the Perspective of Causal Mechanism Identification [19.389891710579022]
  We study the use of large language models (LLMs) as feature enhancers to optimize node representations, which are then used as inputs for graph neural networks (GNNs)<n>Building on the analytical results, we design a plug-and-play optimization module to improve the information transfer between LLM enhancers and GNNs.
  arXiv  Detail & Related papers  (2025-05-13T06:29:25Z)
• How to Make LLMs Strong Node Classifiers? [70.14063765424012]
  Language Models (LMs) are challenging the dominance of domain-specific models, such as Graph Neural Networks (GNNs) and Graph Transformers (GTs)<n>We propose a novel approach that empowers off-the-shelf LMs to achieve performance comparable to state-of-the-art (SOTA) GNNs on node classification tasks.
  arXiv  Detail & Related papers  (2024-10-03T08:27:54Z)
• Cross-Modal Learning for Chemistry Property Prediction: Large Language Models Meet Graph Machine Learning [0.0]
  We introduce a Multi-Modal Fusion (MMF) framework that harnesses the analytical prowess of Graph Neural Networks (GNNs) and the linguistic generative and predictive abilities of Large Language Models (LLMs) Our framework combines the effectiveness of GNNs in modeling graph-structured data with the zero-shot and few-shot learning capabilities of LLMs, enabling improved predictions while reducing the risk of overfitting.
  arXiv  Detail & Related papers  (2024-08-27T11:10:39Z)
• Proficient Graph Neural Network Design by Accumulating Knowledge on Large Language Models [20.31388126105889]
  DesiGNN is a knowledge-centered framework that converts past model design experiences into structured, fine-grained knowledge priors.<n>By constructing a solid meta-knowledge between unseen graph understanding and known effective architecture patterns, DesiGNN can deliver top-5.77% initial model proposals for unseen datasets within seconds.
  arXiv  Detail & Related papers  (2024-08-13T08:22:01Z)
• LOGIN: A Large Language Model Consulted Graph Neural Network Training Framework [30.54068909225463]
  We aim to streamline the GNN design process and leverage the advantages of Large Language Models (LLMs) to improve the performance of GNNs on downstream tasks. We formulate a new paradigm, coined "LLMs-as-Consultants," which integrates LLMs with GNNs in an interactive manner. We empirically evaluate the effectiveness of LOGIN on node classification tasks across both homophilic and heterophilic graphs.
  arXiv  Detail & Related papers  (2024-05-22T18:17:20Z)
• ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graph [142.42275983201978]
  We propose a subgraph-aware self-attention mechanism to imitate the GNN for performing structured reasoning. We also adopt an adaptation tuning strategy to adapt the model parameters with 20,000 subgraphs with synthesized questions. Experiments show that ReasoningLM surpasses state-of-the-art models by a large margin, even with fewer updated parameters and less training data.
  arXiv  Detail & Related papers  (2023-12-30T07:18:54Z)
• Unleashing the potential of GNNs via Bi-directional Knowledge Transfer [58.64807174714959]
  Bi-directional Knowledge Transfer (BiKT) is a plug-and-play approach to unleash the potential of the feature transformation operations without modifying the original architecture. BiKT brings up to 0.5% - 4% performance gain over the original GNN, which means a boosted GNN is obtained.
  arXiv  Detail & Related papers  (2023-10-26T04:11:49Z)
• Mastering Symbolic Operations: Augmenting Language Models with Compiled Neural Networks [48.14324895100478]
  "Neural architecture" integrates compiled neural networks (CoNNs) into a standard transformer. CoNNs are neural modules designed to explicitly encode rules through artificially generated attention weights. Experiments demonstrate superiority of our approach over existing techniques in terms of length generalization, efficiency, and interpretability for symbolic operations.
  arXiv  Detail & Related papers  (2023-04-04T09:50:07Z)
• Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPs [71.93227401463199]
  This paper pinpoints the major source of GNNs' performance gain to their intrinsic capability, by introducing an intermediate model class dubbed as P(ropagational)MLP. We observe that PMLPs consistently perform on par with (or even exceed) their GNN counterparts, while being much more efficient in training.
  arXiv  Detail & Related papers  (2022-12-18T08:17:32Z)
• On the Intrinsic Structures of Spiking Neural Networks [66.57589494713515]
  Recent years have emerged a surge of interest in SNNs owing to their remarkable potential to handle time-dependent and event-driven data. There has been a dearth of comprehensive studies examining the impact of intrinsic structures within spiking computations. This work delves deep into the intrinsic structures of SNNs, by elucidating their influence on the expressivity of SNNs.
  arXiv  Detail & Related papers  (2022-06-21T09:42:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.