GIST: Distributed Training for Large-Scale Graph Convolutional Networks

• URL: http://arxiv.org/abs/2102.10424v1
• Date: Sat, 20 Feb 2021 19:25:38 GMT
• Title: GIST: Distributed Training for Large-Scale Graph Convolutional Networks
• Authors: Cameron R. Wolfe, Jingkang Yang, Arindam Chowdhury, Chen Dun, Artun Bayer, Santiago Segarra, Anastasios Kyrillidis
• Abstract summary: GIST is a hybrid layer and graph sampling method, which disjointly partitions the global model into several, smaller sub-GCNs. This distributed framework improves model performance and significantly decreases wall-clock training time. GIST seeks to enable large-scale GCN experimentation with the goal of bridging the existing gap in scale between graph machine learning and deep learning.
• Score: 18.964079367668262
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The graph convolutional network (GCN) is a go-to solution for machine learning on graphs, but its training is notoriously difficult to scale in terms of both the size of the graph and the number of model parameters. These limitations are in stark contrast to the increasing scale (in data size and model size) of experiments in deep learning research. In this work, we propose GIST, a novel distributed approach that enables efficient training of wide (overparameterized) GCNs on large graphs. GIST is a hybrid layer and graph sampling method, which disjointly partitions the global model into several, smaller sub-GCNs that are independently trained across multiple GPUs in parallel. This distributed framework improves model performance and significantly decreases wall-clock training time. GIST seeks to enable large-scale GCN experimentation with the goal of bridging the existing gap in scale between graph machine learning and deep learning.

Related papers

• Slicing Input Features to Accelerate Deep Learning: A Case Study with Graph Neural Networks [0.24578723416255746]
  This paper introduces SliceGCN, a feature-sliced distributed large-scale graph learning method. It aims to avoid accuracy loss typically associated with mini-batch training and to reduce inter- GPU communication. Experiments were conducted on six node classification datasets, yielding some interesting analytical results.
  arXiv  Detail & Related papers  (2024-08-21T10:18:41Z)
• Do We Really Need Graph Convolution During Training? Light Post-Training Graph-ODE for Efficient Recommendation [34.93725892725111]
  Graph convolution networks (GCNs) in training recommender systems (RecSys) have been persistent concerns. This paper presents a critical examination of the necessity of graph convolutions during the training phase. We introduce an innovative alternative: the Light Post-Training Graph Ordinary-Differential-Equation (LightGODE)
  arXiv  Detail & Related papers  (2024-07-26T17:59:32Z)
• Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication [100.51884192970499]
  GNNs are a powerful family of neural networks for learning over graphs. scaling GNNs either by deepening or widening suffers from prevalent issues of unhealthy gradients, over-smoothening, information squashing. We propose not to deepen or widen current GNNs, but instead present a data-centric perspective of model soups tailored for GNNs.
  arXiv  Detail & Related papers  (2023-06-18T03:33:46Z)
• Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling [60.0185734837814]
  Graph neural networks (GNNs) have found extensive applications in learning from graph data. To bolster the generalization capacity of GNNs, it has become customary to augment training graph structures with techniques like graph augmentations. This study introduces the concept of Mixture-of-Experts (MoE) to GNNs, with the aim of augmenting their capacity to adapt to a diverse range of training graph structures.
  arXiv  Detail & Related papers  (2023-04-06T01:09:36Z)
• Scalable Graph Convolutional Network Training on Distributed-Memory Systems [5.169989177779801]
  Graph Convolutional Networks (GCNs) are extensively utilized for deep learning on graphs. Since the convolution operation on graphs induces irregular memory access patterns, designing a memory- and communication-efficient parallel algorithm for GCN training poses unique challenges. We propose a highly parallel training algorithm that scales to large processor counts.
  arXiv  Detail & Related papers  (2022-12-09T17:51:13Z)
• A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking [124.21408098724551]
  Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs) We present a new ensembling training manner, named EnGCN, to address the existing issues. Our proposed method has achieved new state-of-the-art (SOTA) performance on large-scale datasets.
  arXiv  Detail & Related papers  (2022-10-14T03:43:05Z)
• Scaling R-GCN Training with Graph Summarization [71.06855946732296]
  Training of Relation Graph Convolutional Networks (R-GCN) does not scale well with the size of the graph. In this work, we experiment with the use of graph summarization techniques to compress the graph. We obtain reasonable results on the AIFB, MUTAG and AM datasets.
  arXiv  Detail & Related papers  (2022-03-05T00:28:43Z)
• GraphTheta: A Distributed Graph Neural Network Learning System With Flexible Training Strategy [5.466414428765544]
  We present a new distributed graph learning system GraphTheta. It supports multiple training strategies and enables efficient and scalable learning on big graphs. This work represents the largest edge-attributed GNN learning task conducted on a billion-scale network in the literature.
  arXiv  Detail & Related papers  (2021-04-21T14:51:33Z)
• Revisiting Graph based Collaborative Filtering: A Linear Residual Graph Convolutional Network Approach [55.44107800525776]
  Graph Convolutional Networks (GCNs) are state-of-the-art graph based representation learning models. In this paper, we revisit GCN based Collaborative Filtering (CF) based Recommender Systems (RS) We show that removing non-linearities would enhance recommendation performance, consistent with the theories in simple graph convolutional networks. We propose a residual network structure that is specifically designed for CF with user-item interaction modeling.
  arXiv  Detail & Related papers  (2020-01-28T04:41:25Z)
• Graph Ordering: Towards the Optimal by Learning [69.72656588714155]
  Graph representation learning has achieved a remarkable success in many graph-based applications, such as node classification, prediction, and community detection. However, for some kind of graph applications, such as graph compression and edge partition, it is very hard to reduce them to some graph representation learning tasks. In this paper, we propose to attack the graph ordering problem behind such applications by a novel learning approach.
  arXiv  Detail & Related papers  (2020-01-18T09:14:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.