TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs

• URL: http://arxiv.org/abs/2308.13490v3
• Date: Tue, 5 Dec 2023 22:36:34 GMT
• Title: TpuGraphs: A Performance Prediction Dataset on Large Tensor Computational Graphs
• Authors: Phitchaya Mangpo Phothilimthana, Sami Abu-El-Haija, Kaidi Cao, Bahare Fatemi, Mike Burrows, Charith Mendis, Bryan Perozzi
• Abstract summary: This paper introduces TpuGraphs, a performance prediction dataset on full tensor programs. Each graph in the dataset represents the main computation of a machine learning workload. TpuGraphs provides 25x more graphs than the largest graph property prediction dataset.
• Score: 24.790481918123103
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Precise hardware performance models play a crucial role in code optimizations. They can assist compilers in making heuristic decisions or aid autotuners in identifying the optimal configuration for a given program. For example, the autotuner for XLA, a machine learning compiler, discovered 10-20% speedup on state-of-the-art models serving substantial production traffic at Google. Although there exist a few datasets for program performance prediction, they target small sub-programs such as basic blocks or kernels. This paper introduces TpuGraphs, a performance prediction dataset on full tensor programs, represented as computational graphs, running on Tensor Processing Units (TPUs). Each graph in the dataset represents the main computation of a machine learning workload, e.g., a training epoch or an inference step. Each data sample contains a computational graph, a compilation configuration, and the execution time of the graph when compiled with the configuration. The graphs in the dataset are collected from open-source machine learning programs, featuring popular model architectures, e.g., ResNet, EfficientNet, Mask R-CNN, and Transformer. TpuGraphs provides 25x more graphs than the largest graph property prediction dataset (with comparable graph sizes), and 770x larger graphs on average compared to existing performance prediction datasets on machine learning programs. This graph-level prediction task on large graphs introduces new challenges in learning, ranging from scalability, training efficiency, to model quality.

Related papers

• SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning [131.04781590452308]
  We present SimTeG, a frustratingly Simple approach for Textual Graph learning. We first perform supervised parameter-efficient fine-tuning (PEFT) on a pre-trained LM on the downstream task. We then generate node embeddings using the last hidden states of finetuned LM.
  arXiv  Detail & Related papers  (2023-08-03T07:00:04Z)
• Sparsity exploitation via discovering graphical models in multi-variate time-series forecasting [1.2762298148425795]
  We propose a decoupled training method, which includes a graph generating module and a GNNs forecasting module. First, we use Graphical Lasso (or GraphLASSO) to directly exploit the sparsity pattern from data to build graph structures. Second, we fit these graph structures and the input data into a Graph Convolutional Recurrent Network (GCRN) to train a forecasting model.
  arXiv  Detail & Related papers  (2023-06-29T16:48:00Z)
• Learning Large Graph Property Prediction via Graph Segment Training [61.344814074335304]
  We propose a general framework that allows learning large graph property prediction with a constant memory footprint. We refine the GST paradigm by introducing a historical embedding table to efficiently obtain embeddings for segments not sampled for backpropagation. Our experiments show that GST-EFD is both memory-efficient and fast, while offering a slight boost on test accuracy over a typical full graph training regime.
  arXiv  Detail & Related papers  (2023-05-21T02:53:25Z)
• Distributed Graph Embedding with Information-Oriented Random Walks [16.290803469068145]
  Graph embedding maps graph nodes to low-dimensional vectors, and is widely adopted in machine learning tasks. We present a general-purpose, distributed, information-centric random walk-based graph embedding framework, DistGER, which can scale to embed billion-edge graphs. D DistGER exhibits 2.33x-129x acceleration, 45% reduction in cross-machines communication, and > 10% effectiveness improvement in downstream tasks.
  arXiv  Detail & Related papers  (2023-03-28T03:11:21Z)
• Scaling R-GCN Training with Graph Summarization [71.06855946732296]
  Training of Relation Graph Convolutional Networks (R-GCN) does not scale well with the size of the graph. In this work, we experiment with the use of graph summarization techniques to compress the graph. We obtain reasonable results on the AIFB, MUTAG and AM datasets.
  arXiv  Detail & Related papers  (2022-03-05T00:28:43Z)
• Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Prediction [123.20238648121445]
  We propose a new self-supervised learning framework, Graph Information Aided Node feature exTraction (GIANT) GIANT makes use of the eXtreme Multi-label Classification (XMC) formalism, which is crucial for fine-tuning the language model based on graph information. We demonstrate the superior performance of GIANT over the standard GNN pipeline on Open Graph Benchmark datasets.
  arXiv  Detail & Related papers  (2021-10-29T19:55:12Z)
• Boosting Graph Embedding on a Single GPU [3.093890460224435]
  We present GOSH, a GPU-based tool for embedding large-scale graphs with minimum hardware constraints. It employs a novel graph coarsening algorithm to enhance the impact of updates and minimize the work for embedding. It also incorporates a decomposition schema that enables any arbitrarily large graph to be embedded with a single GPU.
  arXiv  Detail & Related papers  (2021-10-19T15:25:04Z)
• Graph Contrastive Learning Automated [94.41860307845812]
  Graph contrastive learning (GraphCL) has emerged with promising representation learning performance. The effectiveness of GraphCL hinges on ad-hoc data augmentations, which have to be manually picked per dataset. This paper proposes a unified bi-level optimization framework to automatically, adaptively and dynamically select data augmentations when performing GraphCL on specific graph data.
  arXiv  Detail & Related papers  (2021-06-10T16:35:27Z)
• Learning Massive Graph Embeddings on a Single Machine [11.949017733445624]
  A graph embedding is a fixed length vector representation for each node (and/or edge-type) in a graph. Current systems for learning the embeddings of large-scale graphs are bottlenecked by data movement. We propose Gaius, a system for efficient training of graph embeddings.
  arXiv  Detail & Related papers  (2021-01-20T23:17:31Z)
• Understanding Coarsening for Embedding Large-Scale Graphs [3.6739949215165164]
  Proper analysis of graphs with Machine Learning (ML) algorithms has the potential to yield far-reaching insights into many areas of research and industry. The irregular structure of graph data constitutes an obstacle for running ML tasks on graphs. We analyze the impact of the coarsening quality on the embedding performance both in terms of speed and accuracy.
  arXiv  Detail & Related papers  (2020-09-10T15:06:33Z)
• Scaling Graph Neural Networks with Approximate PageRank [64.92311737049054]
  We present the PPRGo model which utilizes an efficient approximation of information diffusion in GNNs. In addition to being faster, PPRGo is inherently scalable, and can be trivially parallelized for large datasets like those found in industry settings. We show that training PPRGo and predicting labels for all nodes in this graph takes under 2 minutes on a single machine, far outpacing other baselines on the same graph.
  arXiv  Detail & Related papers  (2020-07-03T09:30:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.