Unleashing the potential of GNNs via Bi-directional Knowledge Transfer

• URL: http://arxiv.org/abs/2310.17132v1
• Date: Thu, 26 Oct 2023 04:11:49 GMT
• Title: Unleashing the potential of GNNs via Bi-directional Knowledge Transfer
• Authors: Shuai Zheng, Zhizhe Liu, Zhenfeng Zhu, Xingxing Zhang, Jianxin Li, and Yao Zhao
• Abstract summary: Bi-directional Knowledge Transfer (BiKT) is a plug-and-play approach to unleash the potential of the feature transformation operations without modifying the original architecture. BiKT brings up to 0.5% - 4% performance gain over the original GNN, which means a boosted GNN is obtained.
• Score: 58.64807174714959
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Based on the message-passing paradigm, there has been an amount of research proposing diverse and impressive feature propagation mechanisms to improve the performance of GNNs. However, less focus has been put on feature transformation, another major operation of the message-passing framework. In this paper, we first empirically investigate the performance of the feature transformation operation in several typical GNNs. Unexpectedly, we notice that GNNs do not completely free up the power of the inherent feature transformation operation. By this observation, we propose the Bi-directional Knowledge Transfer (BiKT), a plug-and-play approach to unleash the potential of the feature transformation operations without modifying the original architecture. Taking the feature transformation operation as a derived representation learning model that shares parameters with the original GNN, the direct prediction by this model provides a topological-agnostic knowledge feedback that can further instruct the learning of GNN and the feature transformations therein. On this basis, BiKT not only allows us to acquire knowledge from both the GNN and its derived model but promotes each other by injecting the knowledge into the other. In addition, a theoretical analysis is further provided to demonstrate that BiKT improves the generalization bound of the GNNs from the perspective of domain adaption. An extensive group of experiments on up to 7 datasets with 5 typical GNNs demonstrates that BiKT brings up to 0.5% - 4% performance gain over the original GNN, which means a boosted GNN is obtained. Meanwhile, the derived model also shows a powerful performance to compete with or even surpass the original GNN, enabling us to flexibly apply it independently to some other specific downstream tasks.

Related papers

• Learning Invariant Representations of Graph Neural Networks via Cluster Generalization [58.68231635082891]
  Graph neural networks (GNNs) have become increasingly popular in modeling graph-structured data. In this paper, we experimentally find that the performance of GNNs drops significantly when the structure shift happens. We propose the Cluster Information Transfer (CIT) mechanism, which can learn invariant representations for GNNs.
  arXiv  Detail & Related papers  (2024-03-06T10:36:56Z)
• AdapterGNN: Parameter-Efficient Fine-Tuning Improves Generalization in GNNs [2.69499085779099]
  We present a comprehensive comparison of PEFT techniques for graph neural networks (GNNs) We propose a novel PEFT method specifically designed for GNNs, called AdapterGNN. We show that AdapterGNN achieves higher performance than other PEFT methods and is the only one consistently surpassing full fine-tuning.
  arXiv  Detail & Related papers  (2023-04-19T12:00:15Z)
• Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPs [71.93227401463199]
  This paper pinpoints the major source of GNNs' performance gain to their intrinsic capability, by introducing an intermediate model class dubbed as P(ropagational)MLP. We observe that PMLPs consistently perform on par with (or even exceed) their GNN counterparts, while being much more efficient in training.
  arXiv  Detail & Related papers  (2022-12-18T08:17:32Z)
• ReFactorGNNs: Revisiting Factorisation-based Models from a Message-Passing Perspective [42.845783579293]
  We bridge the gap between Factorisation-based Models (FMs) and Graph Neural Networks (GNNs) by proposing ReFactorGNNs. We show how FMs can be cast as GNNs by reformulating the gradient descent procedure as message-passing operations. Our ReFactorGNNs achieve comparable transductive performance to FMs, and state-of-the-art inductive performance while using an order of magnitude fewer parameters.
  arXiv  Detail & Related papers  (2022-07-20T15:39:30Z)
• Investigating Transfer Learning in Graph Neural Networks [2.320417845168326]
  Graph neural networks (GNNs) build on the success of deep learning models by extending them for use in graph spaces. transfer learning has proven extremely successful for traditional deep learning problems: resulting in faster training and improved performance. This research demonstrates that transfer learning is effective with GNNs, and describes how source tasks and the choice of GNN impact the ability to learn generalisable knowledge.
  arXiv  Detail & Related papers  (2022-02-01T20:33:15Z)
• Orthogonal Graph Neural Networks [53.466187667936026]
  Graph neural networks (GNNs) have received tremendous attention due to their superiority in learning node representations. stacking more convolutional layers significantly decreases the performance of GNNs. We propose a novel Ortho-GConv, which could generally augment the existing GNN backbones to stabilize the model training and improve the model's generalization performance.
  arXiv  Detail & Related papers  (2021-09-23T12:39:01Z)
• Interpreting and Unifying Graph Neural Networks with An Optimization Framework [47.44773358082203]
  Graph Neural Networks (GNNs) have received considerable attention on graph-structured data learning. In this paper, we establish a surprising connection between different propagation mechanisms with a unified optimization problem. Our proposed unified optimization framework, summarizing the commonalities between several of the most representative GNNs, opens up new opportunities for flexibly designing new GNNs.
  arXiv  Detail & Related papers  (2021-01-28T08:06:02Z)
• The Surprising Power of Graph Neural Networks with Random Node Initialization [54.4101931234922]
  Graph neural networks (GNNs) are effective models for representation learning on relational data. Standard GNNs are limited in their expressive power, as they cannot distinguish beyond the capability of the Weisfeiler-Leman graph isomorphism. In this work, we analyze the expressive power of GNNs with random node (RNI) We prove that these models are universal, a first such result for GNNs not relying on computationally demanding higher-order properties.
  arXiv  Detail & Related papers  (2020-10-02T19:53:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.