Learning Massive Graph Embeddings on a Single Machine
- URL: http://arxiv.org/abs/2101.08358v1
- Date: Wed, 20 Jan 2021 23:17:31 GMT
- Title: Learning Massive Graph Embeddings on a Single Machine
- Authors: Jason Mohoney, Roger Waleffe, Yiheng Xu, Theodoros Rekatsinas,
Shivaram Venkataraman
- Abstract summary: A graph embedding is a fixed length vector representation for each node (and/or edge-type) in a graph.
Current systems for learning the embeddings of large-scale graphs are bottlenecked by data movement.
We propose Gaius, a system for efficient training of graph embeddings.
- Score: 11.949017733445624
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We propose a new framework for computing the embeddings of large-scale graphs
on a single machine. A graph embedding is a fixed length vector representation
for each node (and/or edge-type) in a graph and has emerged as the de-facto
approach to apply modern machine learning on graphs. We identify that current
systems for learning the embeddings of large-scale graphs are bottlenecked by
data movement, which results in poor resource utilization and inefficient
training. These limitations require state-of-the-art systems to distribute
training across multiple machines. We propose Gaius, a system for efficient
training of graph embeddings that leverages partition caching and buffer-aware
data orderings to minimize disk access and interleaves data movement with
computation to maximize utilization. We compare Gaius against two
state-of-the-art industrial systems on a diverse array of benchmarks. We
demonstrate that Gaius achieves the same level of accuracy but is up to one
order-of magnitude faster. We also show that Gaius can scale training to
datasets an order of magnitude beyond a single machine's GPU and CPU memory
capacity, enabling training of configurations with more than a billion edges
and 550GB of total parameters on a single AWS P3.2xLarge instance.
Related papers
- GraphScale: A Framework to Enable Machine Learning over Billion-node Graphs [6.418397511692011]
We propose a unified framework for both supervised and unsupervised learning to store and process large graph data distributedly.
The key insight in our design is the separation of workers who store data and those who perform the training.
Our experiments show that GraphScale outperforms state-of-the-art methods for distributed training of both GNNs and node embeddings.
arXiv Detail & Related papers (2024-07-22T08:09:36Z) - Communication-Free Distributed GNN Training with Vertex Cut [63.22674903170953]
CoFree-GNN is a novel distributed GNN training framework that significantly speeds up the training process by implementing communication-free training.
We demonstrate that CoFree-GNN speeds up the GNN training process by up to 10 times over the existing state-of-the-art GNN training approaches.
arXiv Detail & Related papers (2023-08-06T21:04:58Z) - Learning Large Graph Property Prediction via Graph Segment Training [61.344814074335304]
We propose a general framework that allows learning large graph property prediction with a constant memory footprint.
We refine the GST paradigm by introducing a historical embedding table to efficiently obtain embeddings for segments not sampled for backpropagation.
Our experiments show that GST-EFD is both memory-efficient and fast, while offering a slight boost on test accuracy over a typical full graph training regime.
arXiv Detail & Related papers (2023-05-21T02:53:25Z) - Distributed Graph Embedding with Information-Oriented Random Walks [16.290803469068145]
Graph embedding maps graph nodes to low-dimensional vectors, and is widely adopted in machine learning tasks.
We present a general-purpose, distributed, information-centric random walk-based graph embedding framework, DistGER, which can scale to embed billion-edge graphs.
D DistGER exhibits 2.33x-129x acceleration, 45% reduction in cross-machines communication, and > 10% effectiveness improvement in downstream tasks.
arXiv Detail & Related papers (2023-03-28T03:11:21Z) - Scalable Graph Convolutional Network Training on Distributed-Memory
Systems [5.169989177779801]
Graph Convolutional Networks (GCNs) are extensively utilized for deep learning on graphs.
Since the convolution operation on graphs induces irregular memory access patterns, designing a memory- and communication-efficient parallel algorithm for GCN training poses unique challenges.
We propose a highly parallel training algorithm that scales to large processor counts.
arXiv Detail & Related papers (2022-12-09T17:51:13Z) - A Comprehensive Study on Large-Scale Graph Training: Benchmarking and
Rethinking [124.21408098724551]
Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs)
We present a new ensembling training manner, named EnGCN, to address the existing issues.
Our proposed method has achieved new state-of-the-art (SOTA) performance on large-scale datasets.
arXiv Detail & Related papers (2022-10-14T03:43:05Z) - Scaling R-GCN Training with Graph Summarization [71.06855946732296]
Training of Relation Graph Convolutional Networks (R-GCN) does not scale well with the size of the graph.
In this work, we experiment with the use of graph summarization techniques to compress the graph.
We obtain reasonable results on the AIFB, MUTAG and AM datasets.
arXiv Detail & Related papers (2022-03-05T00:28:43Z) - Scalable Graph Embedding LearningOn A Single GPU [18.142879223260785]
We introduce a hybrid CPU-GPU framework that addresses the challenges of learning embedding of large-scale graphs.
We show that our system can scale training to datasets with an order of magnitude greater than a single machine's total memory capacity.
arXiv Detail & Related papers (2021-10-13T19:09:33Z) - Distributed Training of Graph Convolutional Networks using Subgraph
Approximation [72.89940126490715]
We propose a training strategy that mitigates the lost information across multiple partitions of a graph through a subgraph approximation scheme.
The subgraph approximation approach helps the distributed training system converge at single-machine accuracy.
arXiv Detail & Related papers (2020-12-09T09:23:49Z) - Scaling Graph Neural Networks with Approximate PageRank [64.92311737049054]
We present the PPRGo model which utilizes an efficient approximation of information diffusion in GNNs.
In addition to being faster, PPRGo is inherently scalable, and can be trivially parallelized for large datasets like those found in industry settings.
We show that training PPRGo and predicting labels for all nodes in this graph takes under 2 minutes on a single machine, far outpacing other baselines on the same graph.
arXiv Detail & Related papers (2020-07-03T09:30:07Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.