An Experimental Comparison of Partitioning Strategies for Distributed
Graph Neural Network Training
- URL: http://arxiv.org/abs/2308.15602v1
- Date: Tue, 29 Aug 2023 19:47:31 GMT
- Title: An Experimental Comparison of Partitioning Strategies for Distributed
Graph Neural Network Training
- Authors: Nikolai Merkel, Daniel Stoll, Ruben Mayer, Hans-Arno Jacobsen
- Abstract summary: Graph neural networks (GNNs) have gained much attention as a growing area of deep learning capable of learning on graph-structured data.
In this paper, we study the effectiveness of graph partitioning for distributed GNN training.
- Score: 14.588837832182026
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Recently, graph neural networks (GNNs) have gained much attention as a
growing area of deep learning capable of learning on graph-structured data.
However, the computational and memory requirements for training GNNs on
large-scale graphs can exceed the capabilities of single machines or GPUs,
making distributed GNN training a promising direction for large-scale GNN
training. A prerequisite for distributed GNN training is to partition the input
graph into smaller parts that are distributed among multiple machines of a
compute cluster. Although graph partitioning has been extensively studied with
regard to graph analytics and graph databases, its effect on GNN training
performance is largely unexplored.
In this paper, we study the effectiveness of graph partitioning for
distributed GNN training. Our study aims to understand how different factors
such as GNN parameters, mini-batch size, graph type, features size, and
scale-out factor influence the effectiveness of graph partitioning. We conduct
experiments with two different GNN systems using vertex and edge partitioning.
We found that graph partitioning is a crucial pre-processing step that can
heavily reduce the training time and memory footprint. Furthermore, our results
show that invested partitioning time can be amortized by reduced GNN training,
making it a relevant optimization.
Related papers
- CATGNN: Cost-Efficient and Scalable Distributed Training for Graph Neural Networks [7.321893519281194]
Existing distributed systems load the entire graph in memory for graph partitioning.
We propose CATGNN, a cost-efficient and scalable distributed GNN training system.
We also propose a novel streaming partitioning algorithm named SPRING for distributed GNN training.
arXiv Detail & Related papers (2024-04-02T20:55:39Z) - GLISP: A Scalable GNN Learning System by Exploiting Inherent Structural
Properties of Graphs [5.410321469222541]
We propose GLISP, a sampling based GNN learning system for industrial scale graphs.
GLISP consists of three core components: graph partitioner, graph sampling service and graph inference engine.
Experiments show that GLISP achieves up to $6.53times$ and $70.77times$ speedups over existing GNN systems for training and inference tasks.
arXiv Detail & Related papers (2024-01-06T02:59:24Z) - Learning to Reweight for Graph Neural Network [63.978102332612906]
Graph Neural Networks (GNNs) show promising results for graph tasks.
Existing GNNs' generalization ability will degrade when there exist distribution shifts between testing and training graph data.
We propose a novel nonlinear graph decorrelation method, which can substantially improve the out-of-distribution generalization ability.
arXiv Detail & Related papers (2023-12-19T12:25:10Z) - Distributed Graph Neural Network Training: A Survey [51.77035975191926]
Graph neural networks (GNNs) are a type of deep learning models that are trained on graphs and have been successfully applied in various domains.
Despite the effectiveness of GNNs, it is still challenging for GNNs to efficiently scale to large graphs.
As a remedy, distributed computing becomes a promising solution of training large-scale GNNs.
arXiv Detail & Related papers (2022-11-01T01:57:00Z) - Training Graph Neural Networks on Growing Stochastic Graphs [114.75710379125412]
Graph Neural Networks (GNNs) rely on graph convolutions to exploit meaningful patterns in networked data.
We propose to learn GNNs on very large graphs by leveraging the limit object of a sequence of growing graphs, the graphon.
arXiv Detail & Related papers (2022-10-27T16:00:45Z) - A Comprehensive Study on Large-Scale Graph Training: Benchmarking and
Rethinking [124.21408098724551]
Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs)
We present a new ensembling training manner, named EnGCN, to address the existing issues.
Our proposed method has achieved new state-of-the-art (SOTA) performance on large-scale datasets.
arXiv Detail & Related papers (2022-10-14T03:43:05Z) - Characterizing and Understanding Distributed GNN Training on GPUs [2.306379679349986]
Graph neural network (GNN) has been demonstrated to be a powerful model in many domains for its effectiveness in learning over graphs.
To scale GNN training for large graphs, a widely adopted approach is distributed training which accelerates training using multiple computing nodes.
arXiv Detail & Related papers (2022-04-18T03:47:28Z) - BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and
Preprocessing [0.0]
Graph neural networks (GNNs) have extended the success of deep neural networks (DNNs) to non-Euclidean graph data.
Existing systems are inefficient to train large graphs with billions of nodes and edges with GPUs.
This paper proposes BGL, a distributed GNN training system designed to address the bottlenecks with a few key ideas.
arXiv Detail & Related papers (2021-12-16T00:37:37Z) - Increase and Conquer: Training Graph Neural Networks on Growing Graphs [116.03137405192356]
We consider the problem of learning a graphon neural network (WNN) by training GNNs on graphs sampled Bernoulli from the graphon.
Inspired by these results, we propose an algorithm to learn GNNs on large-scale graphs that, starting from a moderate number of nodes, successively increases the size of the graph during training.
arXiv Detail & Related papers (2021-06-07T15:05:59Z) - A Unified Lottery Ticket Hypothesis for Graph Neural Networks [82.31087406264437]
We present a unified GNN sparsification (UGS) framework that simultaneously prunes the graph adjacency matrix and the model weights.
We further generalize the popular lottery ticket hypothesis to GNNs for the first time, by defining a graph lottery ticket (GLT) as a pair of core sub-dataset and sparse sub-network.
arXiv Detail & Related papers (2021-02-12T21:52:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.