SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks

• URL: http://arxiv.org/abs/2106.02743v1
• Date: Fri, 4 Jun 2021 22:20:47 GMT
• Title: SpreadGNN: Serverless Multi-task Federated Learning for Graph Neural Networks
• Authors: Chaoyang He, Emir Ceyani, Keshav Balasubramanian, Murali Annavaram, Salman Avestimehr
• Abstract summary: Graph Neural Networks (GNNs) are the first choice methods for graph machine learning problems. Centralizing a massive amount of real-world graph data for GNN training is prohibitive due to user-side privacy concerns. This work proposes SpreadGNN, a novel multi-task federated training framework.
• Score: 13.965982814292971
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Graph Neural Networks (GNNs) are the first choice methods for graph machine learning problems thanks to their ability to learn state-of-the-art level representations from graph-structured data. However, centralizing a massive amount of real-world graph data for GNN training is prohibitive due to user-side privacy concerns, regulation restrictions, and commercial competition. Federated Learning is the de-facto standard for collaborative training of machine learning models over many distributed edge devices without the need for centralization. Nevertheless, training graph neural networks in a federated setting is vaguely defined and brings statistical and systems challenges. This work proposes SpreadGNN, a novel multi-task federated training framework capable of operating in the presence of partial labels and absence of a central server for the first time in the literature. SpreadGNN extends federated multi-task learning to realistic serverless settings for GNNs, and utilizes a novel optimization algorithm with a convergence guarantee, Decentralized Periodic Averaging SGD (DPA-SGD), to solve decentralized multi-task learning problems. We empirically demonstrate the efficacy of our framework on a variety of non-I.I.D. distributed graph-level molecular property prediction datasets with partial labels. Our results show that SpreadGNN outperforms GNN models trained over a central server-dependent federated learning system, even in constrained topologies. The source code is publicly available at https://github.com/FedML-AI/SpreadGNN

Related papers

• Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication [100.51884192970499]
  GNNs are a powerful family of neural networks for learning over graphs. scaling GNNs either by deepening or widening suffers from prevalent issues of unhealthy gradients, over-smoothening, information squashing. We propose not to deepen or widen current GNNs, but instead present a data-centric perspective of model soups tailored for GNNs.
  arXiv  Detail & Related papers  (2023-06-18T03:33:46Z)
• Lumos: Heterogeneity-aware Federated Graph Learning over Decentralized Devices [19.27111697495379]
  Graph neural networks (GNNs) have been widely deployed in real-world networked applications and systems. We propose the first federated GNN framework called Lumos that supports supervised and unsupervised learning. Based on the constructed tree for each client, a decentralized tree-based GNN trainer is proposed to support versatile training.
  arXiv  Detail & Related papers  (2023-03-01T13:27:06Z)
• Scalable Neural Network Training over Distributed Graphs [45.151244961817454]
  Realworld graph data must often be stored across many machines just because capacity constraints. Network communication is costly and becomes main bottleneck to train GNNs. First framework that can be used to train GNNs at all network decentralization levels.
  arXiv  Detail & Related papers  (2023-02-25T10:42:34Z)
• Distributed Graph Neural Network Training: A Survey [51.77035975191926]
  Graph neural networks (GNNs) are a type of deep learning models that are trained on graphs and have been successfully applied in various domains. Despite the effectiveness of GNNs, it is still challenging for GNNs to efficiently scale to large graphs. As a remedy, distributed computing becomes a promising solution of training large-scale GNNs.
  arXiv  Detail & Related papers  (2022-11-01T01:57:00Z)
• Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks [22.728439336309858]
  We propose a communication-efficient distributed GNN training technique named $textLearn Locally, Correct Globally$ (LLCG) LLCG trains a GNN on its local data by ignoring the dependency between nodes among different machines, then sends the locally trained model to the server for periodic model averaging. We rigorously analyze the convergence of distributed methods with periodic model averaging for training GNNs and show that naively applying periodic model averaging but ignoring the dependency between nodes will suffer from an irreducible residual error.
  arXiv  Detail & Related papers  (2021-11-16T03:07:01Z)
• Wide and Deep Graph Neural Network with Distributed Online Learning [174.8221510182559]
  Graph neural networks (GNNs) are naturally distributed architectures for learning representations from network data. Online learning can be leveraged to retrain GNNs at testing time to overcome this issue. This paper develops the Wide and Deep GNN (WD-GNN), a novel architecture that can be updated with distributed online learning mechanisms.
  arXiv  Detail & Related papers  (2021-07-19T23:56:48Z)
• FedGraphNN: A Federated Learning System and Benchmark for Graph Neural Networks [68.64678614325193]
  Graph Neural Network (GNN) research is rapidly growing thanks to the capacity of GNNs to learn representations from graph-structured data. Centralizing a massive amount of real-world graph data for GNN training is prohibitive due to user-side privacy concerns. We introduce FedGraphNN, an open research federated learning system and a benchmark to facilitate GNN-based FL research.
  arXiv  Detail & Related papers  (2021-04-14T22:11:35Z)
• A Unified Lottery Ticket Hypothesis for Graph Neural Networks [82.31087406264437]
  We present a unified GNN sparsification (UGS) framework that simultaneously prunes the graph adjacency matrix and the model weights. We further generalize the popular lottery ticket hypothesis to GNNs for the first time, by defining a graph lottery ticket (GLT) as a pair of core sub-dataset and sparse sub-network.
  arXiv  Detail & Related papers  (2021-02-12T21:52:43Z)
• Wide and Deep Graph Neural Networks with Distributed Online Learning [175.96910854433574]
  Graph neural networks (GNNs) learn representations from network data with naturally distributed architectures. Online learning can be used to retrain GNNs at testing time, overcoming this issue. This paper proposes the Wide and Deep GNN (WD-GNN), a novel architecture that can be easily updated with distributed online learning mechanisms.
  arXiv  Detail & Related papers  (2020-06-11T12:48:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.