Efficient GNN Training Through Structure-Aware Randomized Mini-Batching

• URL: http://arxiv.org/abs/2504.18082v1
• Date: Fri, 25 Apr 2025 05:16:53 GMT
• Title: Efficient GNN Training Through Structure-Aware Randomized Mini-Batching
• Authors: Vignesh Balaji, Christos Kozyrakis, Gal Chechik, Haggai Maron,
• Abstract summary: Graph Neural Networks (GNNs) enable learning on realworld graphs and mini-batch training has emerged as the de facto standard for training GNNs.<n>Existing mini-batching techniques employ randomization schemes to improve accuracy and convergence.<n>We present Community-structure-aware Randomized Mini-batching (COMM-RAND), a novel methodology that bridges the gap between the above extremes.
• Score: 41.7545164294969
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Graph Neural Networks (GNNs) enable learning on realworld graphs and mini-batch training has emerged as the de facto standard for training GNNs because it can scale to very large graphs and improve convergence. Current mini-batch construction policies largely ignore efficiency considerations of GNN training. Specifically, existing mini-batching techniques employ randomization schemes to improve accuracy and convergence. However, these randomization schemes are often agnostic to the structural properties of the graph (for eg. community structure), resulting in highly irregular memory access patterns during GNN training that make suboptimal use of on-chip GPU caches. On the other hand, while deterministic mini-batching based solely on graph structure delivers fast runtime performance, the lack of randomness compromises both the final model accuracy and training convergence speed. In this paper, we present Community-structure-aware Randomized Mini-batching (COMM-RAND), a novel methodology that bridges the gap between the above extremes. COMM-RAND allows practitioners to explore the space between pure randomness and pure graph structural awareness during mini-batch construction, leading to significantly more efficient GNN training with similar accuracy. We evaluated COMM-RAND across four popular graph learning benchmarks. COMM-RAND cuts down GNN training time by up to 2.76x (1.8x on average) while achieving an accuracy that is within 1.79% points (0.42% on average) compared to popular random mini-batching approaches.

Related papers

• Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networks [14.061451788125938]
  We study distributed training of Graph Neural Networks (GNNs) on billion-scale graphs that are partitioned across machines.<n> Efficient training in this setting relies on min-edge-cut partitioning algorithms, which minimize cross-machine communication due to GNN neighborhood sampling.<n>We introduce Armada, a new end-to-end system for distributed GNN training whose key contribution is GREM, a novel min-edge-cut partitioning algorithm that can efficiently scale to large graphs.
  arXiv  Detail & Related papers  (2025-02-25T04:47:39Z)
• Fast Graph Sharpness-Aware Minimization for Enhancing and Accelerating Few-Shot Node Classification [53.727688136434345]
  Graph Neural Networks (GNNs) have shown superior performance in node classification. We present Fast Graph Sharpness-Aware Minimization (FGSAM) that integrates the rapid training of Multi-Layer Perceptrons with the superior performance of GNNs. Our proposed algorithm outperforms the standard SAM with lower computational costs in FSNC tasks.
  arXiv  Detail & Related papers  (2024-10-22T09:33:29Z)
• Entropy Aware Training for Fast and Accurate Distributed GNN [0.0]
  Several distributed frameworks have been developed to scale Graph Neural Networks (GNNs) on billion-size graphs. We develop techniques that reduce training time and improve accuracy. We implement our algorithms on the DistDGL framework and observe that our training techniques scale much better than the existing training approach.
  arXiv  Detail & Related papers  (2023-11-04T13:11:49Z)
• Efficient Heterogeneous Graph Learning via Random Projection [58.4138636866903]
  Heterogeneous Graph Neural Networks (HGNNs) are powerful tools for deep learning on heterogeneous graphs. Recent pre-computation-based HGNNs use one-time message passing to transform a heterogeneous graph into regular-shaped tensors. We propose a hybrid pre-computation-based HGNN, named Random Projection Heterogeneous Graph Neural Network (RpHGNN)
  arXiv  Detail & Related papers  (2023-10-23T01:25:44Z)
• Communication-Free Distributed GNN Training with Vertex Cut [63.22674903170953]
  CoFree-GNN is a novel distributed GNN training framework that significantly speeds up the training process by implementing communication-free training. We demonstrate that CoFree-GNN speeds up the GNN training process by up to 10 times over the existing state-of-the-art GNN training approaches.
  arXiv  Detail & Related papers  (2023-08-06T21:04:58Z)
• Fast and Effective GNN Training through Sequences of Random Path Graphs [20.213843086649014]
  We present GERN, a novel scalable framework for training GNNs in node classification tasks.<n>Our method progressively refines the GNN weights on a sequence of random spanning trees suitably transformed into path graphs.<n>The sparse nature of these path graphs substantially lightens the computational burden of GNN training.
  arXiv  Detail & Related papers  (2023-06-07T23:12:42Z)
• A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking [124.21408098724551]
  Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs) We present a new ensembling training manner, named EnGCN, to address the existing issues. Our proposed method has achieved new state-of-the-art (SOTA) performance on large-scale datasets.
  arXiv  Detail & Related papers  (2022-10-14T03:43:05Z)
• Comprehensive Graph Gradual Pruning for Sparse Training in Graph Neural Networks [52.566735716983956]
  We propose a graph gradual pruning framework termed CGP to dynamically prune GNNs. Unlike LTH-based methods, the proposed CGP approach requires no re-training, which significantly reduces the computation costs. Our proposed strategy greatly improves both training and inference efficiency while matching or even exceeding the accuracy of existing methods.
  arXiv  Detail & Related papers  (2022-07-18T14:23:31Z)
• Scalable Consistency Training for Graph Neural Networks via Self-Ensemble Self-Distillation [13.815063206114713]
  We introduce a novel consistency training method to improve accuracy of graph neural networks (GNNs) For a target node we generate different neighborhood expansions, and distill the knowledge of the average of the predictions to the GNN. Our method approximates the expected prediction of the possible neighborhood samples and practically only requires a few samples.
  arXiv  Detail & Related papers  (2021-10-12T19:24:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.