Community-based Layerwise Distributed Training of Graph Convolutional Networks

• URL: http://arxiv.org/abs/2112.09335v1
• Date: Fri, 17 Dec 2021 05:50:08 GMT
• Title: Community-based Layerwise Distributed Training of Graph Convolutional Networks
• Authors: Hongyi Li, Junxiang Wang, Yongchao Wang, Yue Cheng, and Liang Zhao
• Abstract summary: We propose a parallel and distributed GCN training algorithm based on the Alternating Direction Method of Multipliers (ADMM) Preliminary results demonstrate that our proposed community-based ADMM training algorithm can lead to more than triple speedup.
• Score: 18.96786634170954
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The Graph Convolutional Network (GCN) has been successfully applied to many graph-based applications. Training a large-scale GCN model, however, is still challenging: Due to the node dependency and layer dependency of the GCN architecture, a huge amount of computational time and memory is required in the training process. In this paper, we propose a parallel and distributed GCN training algorithm based on the Alternating Direction Method of Multipliers (ADMM) to tackle the two challenges simultaneously. We first split GCN layers into independent blocks to achieve layer parallelism. Furthermore, we reduce node dependency by dividing the graph into several dense communities such that each of them can be trained with an agent in parallel. Finally, we provide solutions for all subproblems in the community-based ADMM algorithm. Preliminary results demonstrate that our proposed community-based ADMM training algorithm can lead to more than triple speedup while achieving the best performance compared with state-of-the-art methods.

Related papers

• MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Accelerators [3.598994359810843]
  Training Graph convolutional networks (GCNs) on full graphs is challenging. Feature tensors can easily explode the memory and block the communication bandwidth of modern accelerators. Workflow in training GCNs alternates between sparse and dense matrix operations.
  arXiv  Detail & Related papers  (2025-01-03T18:54:46Z)
• Brain-inspired Chaotic Graph Backpropagation for Large-scale Combinatorial Optimization [3.97492577026225]
  Graph neural networks (GNNs) with unsupervised learning can solve large-scale optimization problems (COPs) with efficient time complexity. However, the current mainstream backpropagation-based training algorithms are prone to fall into local minima. We introduce a chaotic training algorithm, i.e. chaotic graph backpropagation (CGBP), which makes the training process not only chaotic but also highly efficient.
  arXiv  Detail & Related papers  (2024-12-13T05:00:57Z)
• FusionLLM: A Decentralized LLM Training System on Geo-distributed GPUs with Adaptive Compression [55.992528247880685]
  Decentralized training faces significant challenges regarding system design and efficiency. We present FusionLLM, a decentralized training system designed and implemented for training large deep neural networks (DNNs) We show that our system and method can achieve 1.45 - 9.39x speedup compared to baseline methods while ensuring convergence.
  arXiv  Detail & Related papers  (2024-10-16T16:13:19Z)
• Ensemble Quadratic Assignment Network for Graph Matching [52.20001802006391]
  Graph matching is a commonly used technique in computer vision and pattern recognition. Recent data-driven approaches have improved the graph matching accuracy remarkably. We propose a graph neural network (GNN) based approach to combine the advantages of data-driven and traditional methods.
  arXiv  Detail & Related papers  (2024-03-11T06:34:05Z)
• T-GAE: Transferable Graph Autoencoder for Network Alignment [79.89704126746204]
  T-GAE is a graph autoencoder framework that leverages transferability and stability of GNNs to achieve efficient network alignment without retraining. Our experiments demonstrate that T-GAE outperforms the state-of-the-art optimization method and the best GNN approach by up to 38.7% and 50.8%, respectively.
  arXiv  Detail & Related papers  (2023-10-05T02:58:29Z)
• GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelism [10.723541176359452]
  Communication is a key bottleneck for distributed graph neural network (GNN) training. GNNPipe is a new approach that scales the distributed full-graph deep GNN training.
  arXiv  Detail & Related papers  (2023-08-19T18:44:14Z)
• A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking [124.21408098724551]
  Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs) We present a new ensembling training manner, named EnGCN, to address the existing issues. Our proposed method has achieved new state-of-the-art (SOTA) performance on large-scale datasets.
  arXiv  Detail & Related papers  (2022-10-14T03:43:05Z)
• Comprehensive Graph Gradual Pruning for Sparse Training in Graph Neural Networks [52.566735716983956]
  We propose a graph gradual pruning framework termed CGP to dynamically prune GNNs. Unlike LTH-based methods, the proposed CGP approach requires no re-training, which significantly reduces the computation costs. Our proposed strategy greatly improves both training and inference efficiency while matching or even exceeding the accuracy of existing methods.
  arXiv  Detail & Related papers  (2022-07-18T14:23:31Z)
• SMGRL: Scalable Multi-resolution Graph Representation Learning [1.878741798127168]
  Graph convolutional networks (GCNs) allow us to learn topologically-aware node embeddings. They are unable to capture long-range dependencies between nodes without adding additional layers. We propose a Scalable Multi-resolution Graph Representation Learning framework that enables us to learn multi-resolution node embeddings efficiently.
  arXiv  Detail & Related papers  (2022-01-29T21:48:48Z)
• Learning Hierarchical Graph Neural Networks for Image Clustering [81.5841862489509]
  We propose a hierarchical graph neural network (GNN) model that learns how to cluster a set of images into an unknown number of identities. Our hierarchical GNN uses a novel approach to merge connected components predicted at each level of the hierarchy to form a new graph at the next level.
  arXiv  Detail & Related papers  (2021-07-03T01:28:42Z)
• MG-GCN: Fast and Effective Learning with Mix-grained Aggregators for Training Large Graph Convolutional Networks [20.07942308916373]
  Graph convolutional networks (GCNs) generate the embeddings of nodes by aggregating the information of their neighbors layer by layer. The high computational and memory cost of GCNs makes it infeasible for training on large graphs. A new model, named Mix-grained GCN (MG-GCN), achieves state-of-the-art performance in terms of accuracy, training speed, convergence speed, and memory cost.
  arXiv  Detail & Related papers  (2020-11-17T14:51:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.