LightHGNN: Distilling Hypergraph Neural Networks into MLPs for $100\times$ Faster Inference

• URL: http://arxiv.org/abs/2402.04296v2
• Date: Sun, 18 Feb 2024 02:20:02 GMT
• Title: LightHGNN: Distilling Hypergraph Neural Networks into MLPs for $100\times$ Faster Inference
• Authors: Yifan Feng, Yihe Luo, Shihui Ying, Yue Gao
• Abstract summary: Hypergraph Networks (HGNNs) have recently attracted much attention and exhibited satisfactory performance due to their superiority in high-order correlation modeling. In this paper, we propose to bridge the gap between the HGNNs and inference-efficient Multi-Layer Perceptron (MLPs) to eliminate the hypergraph dependency of HGNNs.
• Score: 19.383356275847444
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Hypergraph Neural Networks (HGNNs) have recently attracted much attention and exhibited satisfactory performance due to their superiority in high-order correlation modeling. However, it is noticed that the high-order modeling capability of hypergraph also brings increased computation complexity, which hinders its practical industrial deployment. In practice, we find that one key barrier to the efficient deployment of HGNNs is the high-order structural dependencies during inference. In this paper, we propose to bridge the gap between the HGNNs and inference-efficient Multi-Layer Perceptron (MLPs) to eliminate the hypergraph dependency of HGNNs and thus reduce computational complexity as well as improve inference speed. Specifically, we introduce LightHGNN and LightHGNN$^+$ for fast inference with low complexity. LightHGNN directly distills the knowledge from teacher HGNNs to student MLPs via soft labels, and LightHGNN$^+$ further explicitly injects reliable high-order correlations into the student MLPs to achieve topology-aware distillation and resistance to over-smoothing. Experiments on eight hypergraph datasets demonstrate that even without hypergraph dependency, the proposed LightHGNNs can still achieve competitive or even better performance than HGNNs and outperform vanilla MLPs by $16.3$ on average. Extensive experiments on three graph datasets further show the average best performance of our LightHGNNs compared with all other methods. Experiments on synthetic hypergraphs with 5.5w vertices indicate LightHGNNs can run $100\times$ faster than HGNNs, showcasing their ability for latency-sensitive deployments.

Related papers

• Teaching MLPs to Master Heterogeneous Graph-Structured Knowledge for Efficient and Accurate Inference [53.38082028252104]
  We introduce HG2M and HG2M+ to combine both HGNN's superior performance and relational's efficient inference. HG2M directly trains students with node features as input and soft labels from teacher HGNNs as targets. HG2Ms demonstrate a 379.24$times$ speedup in inference over HGNNs on the large-scale IGB-3M-19 dataset.
  arXiv  Detail & Related papers  (2024-11-21T11:39:09Z)
• MAG-GNN: Reinforcement Learning Boosted Graph Neural Network [68.60884768323739]
  A particular line of work proposed subgraph GNNs that use subgraph information to improve GNNs' expressivity and achieved great success. Such effectivity sacrifices the efficiency of GNNs by enumerating all possible subgraphs. We propose Magnetic Graph Neural Network (MAG-GNN), a reinforcement learning (RL) boosted GNN, to solve the problem.
  arXiv  Detail & Related papers  (2023-10-29T20:32:21Z)
• Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication [100.51884192970499]
  GNNs are a powerful family of neural networks for learning over graphs. scaling GNNs either by deepening or widening suffers from prevalent issues of unhealthy gradients, over-smoothening, information squashing. We propose not to deepen or widen current GNNs, but instead present a data-centric perspective of model soups tailored for GNNs.
  arXiv  Detail & Related papers  (2023-06-18T03:33:46Z)
• Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPs [71.93227401463199]
  This paper pinpoints the major source of GNNs' performance gain to their intrinsic capability, by introducing an intermediate model class dubbed as P(ropagational)MLP. We observe that PMLPs consistently perform on par with (or even exceed) their GNN counterparts, while being much more efficient in training.
  arXiv  Detail & Related papers  (2022-12-18T08:17:32Z)
• Teaching Yourself: Graph Self-Distillation on Neighborhood for Node Classification [42.840122801915996]
  We propose a Graph Self-Distillation on Neighborhood (GSDN) framework to reduce the gap between GNNs and Neurals. GSDN infers 75XX faster than existing GNNs and 16X-25X faster than other inference acceleration methods.
  arXiv  Detail & Related papers  (2022-10-05T08:35:34Z)
• MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initialization [51.76758674012744]
  Training graph neural networks (GNNs) on large graphs is complex and extremely time consuming. We propose an embarrassingly simple, yet hugely effective method for GNN training acceleration, called PeerInit.
  arXiv  Detail & Related papers  (2022-09-30T21:33:51Z)
• Equivariant Hypergraph Diffusion Neural Operators [81.32770440890303]
  Hypergraph neural networks (HNNs) using neural networks to encode hypergraphs provide a promising way to model higher-order relations in data. This work proposes a new HNN architecture named ED-HNN, which provably represents any continuous equivariant hypergraph diffusion operators. We evaluate ED-HNN for node classification on nine real-world hypergraph datasets.
  arXiv  Detail & Related papers  (2022-07-14T06:17:00Z)
• Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation [34.676755383361005]
  Graph-less Neural Networks (GLNNs) have no inference graph dependency. We show that GLNNs with competitive performance infer faster than GNNs by 146X-273X and faster than other acceleration methods by 14X-27X. A comprehensive analysis of GLNN shows when and why GLNN can achieve competitive results to Gs and suggests GLNN as a handy choice for latency-constrained applications.
  arXiv  Detail & Related papers  (2021-10-17T05:16:58Z)
• UniGNN: a Unified Framework for Graph and Hypergraph Neural Networks [8.777765815864367]
  Hypergraph, an expressive structure with flexibility to model the higher-order correlations among entities, has recently attracted increasing attention from various research domains. We propose UniGNN, a unified framework for interpreting the message passing process in graph and hypergraph neural networks. Experiments have been conducted to demonstrate the effectiveness of UniGNN on multiple real-world datasets.
  arXiv  Detail & Related papers  (2021-05-03T15:48:34Z)
• Residual Enhanced Multi-Hypergraph Neural Network [26.42547421121713]
  HyperGraph Neural Network (HGNN) is the de-facto method for hypergraph representation learning. We propose the Residual enhanced Multi-Hypergraph Neural Network, which can fuse multi-modal information from each hypergraph effectively.
  arXiv  Detail & Related papers  (2021-05-02T14:53:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.