TIRAMISU: A Polyhedral Compiler for Dense and Sparse Deep Learning

• URL: http://arxiv.org/abs/2005.04091v1
• Date: Thu, 7 May 2020 07:27:08 GMT
• Title: TIRAMISU: A Polyhedral Compiler for Dense and Sparse Deep Learning
• Authors: Riyadh Baghdadi, Abdelkader Nadir Debbagh, Kamel Abdous, Fatima Zohra Benhamida, Alex Renda, Jonathan Elliott Frankle, Michael Carbin and Saman Amarasinghe
• Abstract summary: In this paper, we demonstrate a compiler that can optimize sparse and recurrent neural networks. Our approach at least matches Intel MKL-DNN and in some cases outperforms it by 5x (on multicore- CPU)
• Score: 10.145707529307462
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we demonstrate a compiler that can optimize sparse and recurrent neural networks, both of which are currently outside of the scope of existing neural network compilers (sparse neural networks here stand for networks that can be accelerated with sparse tensor algebra techniques). Our demonstration includes a mapping of sparse and recurrent neural networks to the polyhedral model along with an implementation of our approach in TIRAMISU, our state-of-the-art polyhedral compiler. We evaluate our approach on a set of deep learning benchmarks and compare our results with hand-optimized industrial libraries. Our results show that our approach at least matches Intel MKL-DNN and in some cases outperforms it by 5x (on multicore-CPUs).

Related papers

• Parallel Neural Networks in Golang [0.0]
  This paper describes the design and implementation of parallel neural networks (PNNs) with the novel programming language Golang. Golang and its inherent parallelization support proved very well for parallel neural network simulation by considerable decreased processing times compared to sequential variants.
  arXiv  Detail & Related papers  (2023-04-19T11:56:36Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Learning with Multigraph Convolutional Filters [153.20329791008095]
  We introduce multigraph convolutional neural networks (MGNNs) as stacked and layered structures where information is processed according to an MSP model. We also develop a procedure for tractable computation of filter coefficients in the MGNNs and a low cost method to reduce the dimensionality of the information transferred between layers.
  arXiv  Detail & Related papers  (2022-10-28T17:00:50Z)
• Lipschitz Bound Analysis of Neural Networks [0.0]
  Lipschitz Bound Estimation is an effective method of regularizing deep neural networks to make them robust against adversarial attacks. In this paper, we highlight the significant gap in obtaining a non-trivial Lipschitz bound certificate for Convolutional Neural Networks (CNNs) We also show that unrolling Convolutional layers or Toeplitz matrices can be employed to convert Convolutional Neural Networks (CNNs) to a Fully Connected Network.
  arXiv  Detail & Related papers  (2022-07-14T23:40:22Z)
• Variable Bitrate Neural Fields [75.24672452527795]
  We present a dictionary method for compressing feature grids, reducing their memory consumption by up to 100x. We formulate the dictionary optimization as a vector-quantized auto-decoder problem which lets us learn end-to-end discrete neural representations in a space where no direct supervision is available.
  arXiv  Detail & Related papers  (2022-06-15T17:58:34Z)
• Incorporating Prior Knowledge into Neural Networks through an Implicit Composite Kernel [1.6383321867266318]
  Implicit Composite Kernel (ICK) is a kernel that combines a kernel implicitly defined by a neural network with a second kernel function chosen to model known properties. We demonstrate ICK's superior performance and flexibility on both synthetic and real-world data sets.
  arXiv  Detail & Related papers  (2022-05-15T21:32:44Z)
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch [75.69506249886622]
  Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network.
  arXiv  Detail & Related papers  (2021-02-08T05:55:47Z)
• SparseDNN: Fast Sparse Deep Learning Inference on CPUs [1.6244541005112747]
  We present SparseDNN, a sparse deep learning inference engine targeting CPUs. We show that our sparse code generator can achieve significant speedups over state-of-the-art sparse and dense libraries.
  arXiv  Detail & Related papers  (2021-01-20T03:27:35Z)
• Generalized Leverage Score Sampling for Neural Networks [82.95180314408205]
  Leverage score sampling is a powerful technique that originates from theoretical computer science. In this work, we generalize the results in [Avron, Kapralov, Musco, Musco, Velingker and Zandieh 17] to a broader class of kernels.
  arXiv  Detail & Related papers  (2020-09-21T14:46:01Z)
• PolyScientist: Automatic Loop Transformations Combined with Microkernels for Optimization of Deep Learning Primitives [55.79741270235602]
  We develop a hybrid solution to the development of deep learning kernels. We use the advanced polyhedral technology to automatically tune the outer loops for performance.
  arXiv  Detail & Related papers  (2020-02-06T08:02:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.