Heuristic Methods are Good Teachers to Distill MLPs for Graph Link Prediction

• URL: http://arxiv.org/abs/2504.06193v1
• Date: Tue, 08 Apr 2025 16:35:11 GMT
• Title: Heuristic Methods are Good Teachers to Distill MLPs for Graph Link Prediction
• Authors: Zongyue Qin, Shichang Zhang, Mingxuan Ju, Tong Zhao, Neil Shah, Yizhou Sun,
• Abstract summary: Distilling Graph Neural Networks (GNNs) teachers into Multi-Layer Perceptrons (MLPs) students has emerged as an effective approach to achieve strong performance.<n>However, existing distillation methods only use standard GNNs and overlook alternative teachers such as specialized model for link prediction (GNN4LP) and methods (e.g., common neighbors)<n>This paper first explores the impact of different teachers in GNN-to-MLP distillation, we find that stronger teachers do not always produce stronger students, while weaker methods can teachs to near-GNN performance with drastically reduced training costs
• Score: 61.70012924088756
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Link prediction is a crucial graph-learning task with applications including citation prediction and product recommendation. Distilling Graph Neural Networks (GNNs) teachers into Multi-Layer Perceptrons (MLPs) students has emerged as an effective approach to achieve strong performance and reducing computational cost by removing graph dependency. However, existing distillation methods only use standard GNNs and overlook alternative teachers such as specialized model for link prediction (GNN4LP) and heuristic methods (e.g., common neighbors). This paper first explores the impact of different teachers in GNN-to-MLP distillation. Surprisingly, we find that stronger teachers do not always produce stronger students: MLPs distilled from GNN4LP can underperform those distilled from simpler GNNs, while weaker heuristic methods can teach MLPs to near-GNN performance with drastically reduced training costs. Building on these insights, we propose Ensemble Heuristic-Distilled MLPs (EHDM), which eliminates graph dependencies while effectively integrating complementary signals via a gating mechanism. Experiments on ten datasets show an average 7.93% improvement over previous GNN-to-MLP approaches with 1.95-3.32 times less training time, indicating EHDM is an efficient and effective link prediction method.

Related papers

• Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference [53.38082028252104]
  We introduce HG2M and HG2M+ to combine both HGNN's superior performance and relational's efficient inference. HG2M directly trains students with node features as input and soft labels from teacher HGNNs as targets. HG2Ms demonstrate a 379.24$times$ speedup in inference over HGNNs on the large-scale IGB-3M-19 dataset.
  arXiv  Detail & Related papers  (2024-11-21T11:39:09Z)
• Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillation [56.912354708167534]
  Graph Neural Networks (GNNs) and lightweight Multi-Layer Perceptron (MLPs) GNNto-MLP Knowledge Distillation (KD) proposes to distill knowledge from a well-trained teacher GNN into a student. This paper proposes a simple yet effective Hardness-aware GNN-to-MLP Distillation (HGMD) framework.
  arXiv  Detail & Related papers  (2024-07-20T06:13:00Z)
• A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation [58.813991312803246]
  We propose a Teacher-Free Graph Self-Distillation (TGS) framework that does not require any teacher model or GNNs during both training and inference. TGS enjoys the benefits of graph topology awareness in training but is free from data dependency in inference.
  arXiv  Detail & Related papers  (2024-03-06T05:52:13Z)
• Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs [9.731314045194495]
  We train a student by knowledge distillation from a teacher neural network (GNN) Inspired by GNNs that separate transformation $T$ and propagation $Pi$, we re-frame the KD process as enabling the student to explicitly learn both $T$ and $Pi$. We propose Propagate & Distill (P&D), which propagates the output of the teacher GNN before KD and can be interpreted as an approximate process of the inverse propagation $Pi-1$.
  arXiv  Detail & Related papers  (2023-11-20T13:39:19Z)
• Extracting Low-/High- Frequency Knowledge from Graph Neural Networks and Injecting it into MLPs: An Effective GNN-to-MLP Distillation Framework [36.160251860788314]
  We propose an efficient Full-Frequency GNN-to-MLP (FFG2M) distillation framework. We factorize the knowledge learned by GNNs into low- and high-frequency components in the spectral domain. We identify a potential information drowning problem for existing GNN-to-MLP distillation.
  arXiv  Detail & Related papers  (2023-05-18T06:57:06Z)
• Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPs [71.93227401463199]
  This paper pinpoints the major source of GNNs' performance gain to their intrinsic capability, by introducing an intermediate model class dubbed as P(ropagational)MLP. We observe that PMLPs consistently perform on par with (or even exceed) their GNN counterparts, while being much more efficient in training.
  arXiv  Detail & Related papers  (2022-12-18T08:17:32Z)
• Linkless Link Prediction via Relational Distillation [24.928349760334413]
  Graph Networks (GNNs) have shown exceptional performance in the task of link prediction. Despite their effectiveness, the high latency brought by non-trivial neighborhood data dependency limits GNNs in practical deployments.
  arXiv  Detail & Related papers  (2022-10-11T22:00:43Z)
• MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initialization [51.76758674012744]
  Training graph neural networks (GNNs) on large graphs is complex and extremely time consuming. We propose an embarrassingly simple, yet hugely effective method for GNN training acceleration, called PeerInit.
  arXiv  Detail & Related papers  (2022-09-30T21:33:51Z)
• On Self-Distilling Graph Neural Network [64.00508355508106]
  We propose the first teacher-free knowledge distillation method for GNNs, termed GNN Self-Distillation (GNN-SD) The method is built upon the proposed neighborhood discrepancy rate (NDR), which quantifies the non-smoothness of the embedded graph in an efficient way. We also summarize a generic GNN-SD framework that could be exploited to induce other distillation strategies.
  arXiv  Detail & Related papers  (2020-11-04T12:29:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.