Characterizing and Understanding HGNN Training on GPUs

• URL: http://arxiv.org/abs/2407.11790v2
• Date: Thu, 18 Jul 2024 00:50:49 GMT
• Title: Characterizing and Understanding HGNN Training on GPUs
• Authors: Dengke Han, Mingyu Yan, Xiaochun Ye, Dongrui Fan, Ninghui Sun,
• Abstract summary: Heterogeneous Graph Neural Networks (HGNNs) have been widely adopted in many real-world domains such as recommendation systems and medical analysis. To enhance the efficiency of HGNN training, it is essential to characterize and analyze the execution semantics and patterns within the training process to identify performance bottlenecks.
• Score: 9.955760955770783
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Owing to their remarkable representation capabilities for heterogeneous graph data, Heterogeneous Graph Neural Networks (HGNNs) have been widely adopted in many critical real-world domains such as recommendation systems and medical analysis. Prior to their practical application, identifying the optimal HGNN model parameters tailored to specific tasks through extensive training is a time-consuming and costly process. To enhance the efficiency of HGNN training, it is essential to characterize and analyze the execution semantics and patterns within the training process to identify performance bottlenecks. In this study, we conduct an in-depth quantification and analysis of two mainstream HGNN training scenarios, including single-GPU and multi-GPU distributed training. Based on the characterization results, we disclose the performance bottlenecks and their underlying causes in different HGNN training scenarios and provide optimization guidelines from both software and hardware perspectives.

Related papers

• SiHGNN: Leveraging Properties of Semantic Graphs for Efficient HGNN Acceleration [9.85638913900595]
  Heterogeneous Graph Neural Networks (HGNNs) have expanded graph representation learning to heterogeneous graph fields. Recent studies have demonstrated their superior performance across various applications, including medical analysis and recommendation systems. We propose a lightweight hardware accelerator for HGNNs, called SiHGNN. This accelerator incorporates a tree-based Semantic Graph Builder for efficient semantic graph generation and features a novel Graph Restructurer for optimizing semantic graph layouts.
  arXiv  Detail & Related papers  (2024-08-27T14:20:21Z)
• A Comprehensive Survey on Distributed Training of Graph Neural Networks [59.785830738482474]
  Graph neural networks (GNNs) have been demonstrated to be a powerful algorithmic model in broad application fields. To scale GNN training up for large-scale and ever-growing graphs, the most promising solution is distributed training. The volume of related research on distributed GNN training is exceptionally vast, accompanied by an extraordinarily rapid pace of publication.
  arXiv  Detail & Related papers  (2022-11-10T06:22:12Z)
• Characterizing the Efficiency of Graph Neural Network Frameworks with a Magnifying Glass [10.839902229218577]
  Graph neural networks (GNNs) have received great attention due to their success in various graph-related learning tasks. Recent GNNs have been developed with different graph sampling techniques for mini-batch training of GNNs on large graphs. It is unknown how much the frameworks are 'eco-friendly' from a green computing perspective.
  arXiv  Detail & Related papers  (2022-11-06T04:22:19Z)
• Distributed Graph Neural Network Training: A Survey [51.77035975191926]
  Graph neural networks (GNNs) are a type of deep learning models that are trained on graphs and have been successfully applied in various domains. Despite the effectiveness of GNNs, it is still challenging for GNNs to efficiently scale to large graphs. As a remedy, distributed computing becomes a promising solution of training large-scale GNNs.
  arXiv  Detail & Related papers  (2022-11-01T01:57:00Z)
• A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking [124.21408098724551]
  Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs) We present a new ensembling training manner, named EnGCN, to address the existing issues. Our proposed method has achieved new state-of-the-art (SOTA) performance on large-scale datasets.
  arXiv  Detail & Related papers  (2022-10-14T03:43:05Z)
• Bottleneck Analysis of Dynamic Graph Neural Network Inference on CPU and GPU [3.4214598355901638]
  Dynamic graph neural network (DGNN) is becoming increasingly popular because of its widespread use in capturing dynamic features in the real world. deploying DGNNs on hardware presents additional challenges due to the model complexity, diversity, and the nature of the time dependency. We select eight prevailing DGNNs with different characteristics and profile them on both CPU and GPU.
  arXiv  Detail & Related papers  (2022-10-08T03:41:50Z)
• MentorGNN: Deriving Curriculum for Pre-Training GNNs [61.97574489259085]
  We propose an end-to-end model named MentorGNN that aims to supervise the pre-training process of GNNs across graphs. We shed new light on the problem of domain adaption on relational data (i.e., graphs) by deriving a natural and interpretable upper bound on the generalization error of the pre-trained GNNs.
  arXiv  Detail & Related papers  (2022-08-21T15:12:08Z)
• Characterizing and Understanding Distributed GNN Training on GPUs [2.306379679349986]
  Graph neural network (GNN) has been demonstrated to be a powerful model in many domains for its effectiveness in learning over graphs. To scale GNN training for large graphs, a widely adopted approach is distributed training which accelerates training using multiple computing nodes.
  arXiv  Detail & Related papers  (2022-04-18T03:47:28Z)
• An Adaptive Graph Pre-training Framework for Localized Collaborative Filtering [79.17319280791237]
  We propose an adaptive graph pre-training framework for localized collaborative filtering (ADAPT) ADAPT captures both the common knowledge across different graphs and the uniqueness for each graph. It does not require transferring user/item embeddings, and is able to capture both the common knowledge across different graphs and the uniqueness for each graph.
  arXiv  Detail & Related papers  (2021-12-14T06:53:13Z)
• Binarized Graph Neural Network [65.20589262811677]
  We develop a binarized graph neural network to learn the binary representations of the nodes with binary network parameters. Our proposed method can be seamlessly integrated into the existing GNN-based embedding approaches. Experiments indicate that the proposed binarized graph neural network, namely BGN, is orders of magnitude more efficient in terms of both time and space.
  arXiv  Detail & Related papers  (2020-04-19T09:43:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.