Skyline: Interactive In-Editor Computational Performance Profiling for Deep Neural Network Training

• URL: http://arxiv.org/abs/2008.06798v2
• Date: Thu, 20 Aug 2020 14:57:58 GMT
• Title: Skyline: Interactive In-Editor Computational Performance Profiling for Deep Neural Network Training
• Authors: Geoffrey X. Yu, Tovi Grossman, Gennady Pekhimenko
• Abstract summary: Skyline is an in-editor tool for training a state-of-the-art deep neural network (DNN) It provides interactive performance predictions and visualizations, and directly manipulatable visualizations that, when dragged, mutate the batch size in the code. An exploratory qualitative user study of Skyline produced promising results.
• Score: 24.512629761651535
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training a state-of-the-art deep neural network (DNN) is a computationally-expensive and time-consuming process, which incentivizes deep learning developers to debug their DNNs for computational performance. However, effectively performing this debugging requires intimate knowledge about the underlying software and hardware systems---something that the typical deep learning developer may not have. To help bridge this gap, we present Skyline: a new interactive tool for DNN training that supports in-editor computational performance profiling, visualization, and debugging. Skyline's key contribution is that it leverages special computational properties of DNN training to provide (i) interactive performance predictions and visualizations, and (ii) directly manipulatable visualizations that, when dragged, mutate the batch size in the code. As an in-editor tool, Skyline allows users to leverage these diagnostic features to debug the performance of their DNNs during development. An exploratory qualitative user study of Skyline produced promising results; all the participants found Skyline to be useful and easy to use.

Related papers

• NNsight and NDIF: Democratizing Access to Open-Weight Foundation Model Internals [58.83169560132308]
  We introduce NNsight and NDIF, technologies that work in tandem to enable scientific study of the representations and computations learned by very large neural networks.
  arXiv  Detail & Related papers  (2024-07-18T17:59:01Z)
• Tempo: Confidentiality Preservation in Cloud-Based Neural Network Training [8.187538747666203]
  Cloud deep learning platforms provide cost-effective deep neural network (DNN) training for customers who lack computation resources. Recently, researchers have sought to protect data privacy in deep learning by leveraging CPU trusted execution environments (TEEs) This paper presents Tempo, the first cloud-based deep learning system that cooperates with TEE and distributed GPU.
  arXiv  Detail & Related papers  (2024-01-21T15:57:04Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Designing and Training of Lightweight Neural Networks on Edge Devices using Early Halting in Knowledge Distillation [16.74710649245842]
  This paper presents a novel approach for designing and training lightweight Deep Neural Networks (DNN) on edge devices. The approach considers the available storage, processing speed, and allowable maximum processing time. We introduce a novel early halting technique, which preserves network resources.
  arXiv  Detail & Related papers  (2022-09-30T16:18:24Z)
• Recurrent Bilinear Optimization for Binary Neural Networks [58.972212365275595]
  BNNs neglect the intrinsic bilinear relationship of real-valued weights and scale factors. Our work is the first attempt to optimize BNNs from the bilinear perspective. We obtain robust RBONNs, which show impressive performance over state-of-the-art BNNs on various models and datasets.
  arXiv  Detail & Related papers  (2022-09-04T06:45:33Z)
• Testing Feedforward Neural Networks Training Programs [13.249453757295083]
  Multiple testing techniques are proposed to generate test cases that can expose inconsistencies in the behavior of Deep Neural Networks. These techniques assume implicitly that the training program is bug-free and appropriately configured. We propose TheDeepChecker, an end-to-end property-based debug approach for DNN training programs.
  arXiv  Detail & Related papers  (2022-04-01T20:49:14Z)
• Privacy-Preserving Graph Neural Network Training and Inference as a Cloud Service [15.939214141337803]
  SecGNN is built from a synergy of insights on lightweight cryptography and machine learning techniques. We show that SecGNN achieves comparable training and inference accuracy, with practically affordable performance.
  arXiv  Detail & Related papers  (2022-02-16T02:57:10Z)
• FPGA-optimized Hardware acceleration for Spiking Neural Networks [69.49429223251178]
  This work presents the development of a hardware accelerator for an SNN, with off-line training, applied to an image recognition task. The design targets a Xilinx Artix-7 FPGA, using in total around the 40% of the available hardware resources. It reduces the classification time by three orders of magnitude, with a small 4.5% impact on the accuracy, if compared to its software, full precision counterpart.
  arXiv  Detail & Related papers  (2022-01-18T13:59:22Z)
• Wide and Deep Graph Neural Network with Distributed Online Learning [174.8221510182559]
  Graph neural networks (GNNs) are naturally distributed architectures for learning representations from network data. Online learning can be leveraged to retrain GNNs at testing time to overcome this issue. This paper develops the Wide and Deep GNN (WD-GNN), a novel architecture that can be updated with distributed online learning mechanisms.
  arXiv  Detail & Related papers  (2021-07-19T23:56:48Z)
• exploRNN: Understanding Recurrent Neural Networks through Visual Exploration [6.006493809079212]
  recurrent neural networks (RNNs) are capable of processing sequential data. We propose exploRNN, the first interactively explorable educational visualization for RNNs. We provide an overview of the training process of RNNs at a coarse level, while also allowing detailed inspection of the data-flow within LSTM cells.
  arXiv  Detail & Related papers  (2020-12-09T15:06:01Z)
• Learning to Execute Programs with Instruction Pointer Attention Graph Neural Networks [55.98291376393561]
  Graph neural networks (GNNs) have emerged as a powerful tool for learning software engineering tasks. Recurrent neural networks (RNNs) are well-suited to long sequential chains of reasoning, but they do not naturally incorporate program structure. We introduce a novel GNN architecture, the Instruction Pointer Attention Graph Neural Networks (IPA-GNN), which improves systematic generalization on the task of learning to execute programs.
  arXiv  Detail & Related papers  (2020-10-23T19:12:30Z)
• Boosting Deep Neural Networks with Geometrical Prior Knowledge: A Survey [77.99182201815763]
  Deep Neural Networks (DNNs) achieve state-of-the-art results in many different problem settings. DNNs are often treated as black box systems, which complicates their evaluation and validation. One promising field, inspired by the success of convolutional neural networks (CNNs) in computer vision tasks, is to incorporate knowledge about symmetric geometrical transformations.
  arXiv  Detail & Related papers  (2020-06-30T14:56:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.