No Saved Kaleidosope: an 100% Jitted Neural Network Coding Language with Pythonic Syntax

• URL: http://arxiv.org/abs/2409.11600v1
• Date: Tue, 17 Sep 2024 23:15:39 GMT
• Title: No Saved Kaleidosope: an 100% Jitted Neural Network Coding Language with Pythonic Syntax
• Authors: Augusto Seben da Rosa, Marlon Daniel Angeli, Jorge Aikes Junior, Alef Iury Ferreira, Lucas Rafael Gris, Anderson da Silva Soares, Arnaldo Candido Junior, Frederico Santos de Oliveira, Gabriel Trevisan Damke, Rafael Teixeira Sousa,
• Abstract summary: We developed a jitted compiler for training Artificial Neural Networks using C++, LLVM and Cuda. It features object-oriented characteristics, strong typing, parallel workers for data pre-processing, pythonic syntax for expressions, PyTorch like model declaration and Automatic Differentiation.
• Score: 0.8408735228878615
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We developed a jitted compiler for training Artificial Neural Networks using C++, LLVM and Cuda. It features object-oriented characteristics, strong typing, parallel workers for data pre-processing, pythonic syntax for expressions, PyTorch like model declaration and Automatic Differentiation. We implement the mechanisms of cache and pooling in order to manage VRAM, cuBLAS for high performance matrix multiplication and cuDNN for convolutional layers. Our experiments with Residual Convolutional Neural Networks on ImageNet, we reach similar speed but degraded performance. Also, the GRU network experiments show similar accuracy, but our compiler have degraded speed in that task. However, our compiler demonstrates promising results at the CIFAR-10 benchmark, in which we reach the same performance and about the same speed as PyTorch. We make the code publicly available at: https://github.com/NoSavedDATA/NoSavedKaleidoscope

Related papers

• Nerva: a Truly Sparse Implementation of Neural Networks [16.29955529463831]
  Nerva is a fast neural network library under development in C++. It supports sparsity by using the sparse matrix operations of Intel's Math Kernel Library.
  arXiv  Detail & Related papers  (2024-07-24T17:13:31Z)
• iSpLib: A Library for Accelerating Graph Neural Networks using Auto-tuned Sparse Operations [1.3030767447016454]
  iSpLib is a PyTorch-based C++ library equipped with auto-tuned sparse operations. We demonstrate that iSpLib obtains up to 27x overall training speedup compared to the equivalent PyTorch 2.1.0 and PyTorch Geometric 2.4.0 implementations on the CPU.
  arXiv  Detail & Related papers  (2024-03-21T21:56:44Z)
• PyGim: An Efficient Graph Neural Network Library for Real Processing-In-Memory Architectures [10.047157906258196]
  We introduce PyGim, an efficient ML library that accelerates Graph Neural Networks on real PIM systems. We provide hybrid GNN execution, in which the compute-intensive and memory-intensive kernels are executed in processor-centric and memory-centric systems. We extensively evaluate PyGim on a real-world PIM system with 1992 PIM cores using emerging GNN models, and demonstrate that it outperforms its state-of-the-art CPU counterpart on Intel Xeon by on average 3.04x.
  arXiv  Detail & Related papers  (2024-02-26T16:52:35Z)
• Comparing neural network training performance between Elixir and Python [0.9023847175654603]
  Python has made a name for itself as one of the main programming languages. In February 2021, Jos'e Valim and Sean Moriarity published the first version of the Numerical Elixir (Nx) library.
  arXiv  Detail & Related papers  (2022-10-25T11:57:14Z)
• PARTIME: Scalable and Parallel Processing Over Time with Deep Neural Networks [68.96484488899901]
  We present PARTIME, a library designed to speed up neural networks whenever data is continuously streamed over time. PARTIME starts processing each data sample at the time in which it becomes available from the stream. Experiments are performed in order to empirically compare PARTIME with classic non-parallel neural computations in online learning.
  arXiv  Detail & Related papers  (2022-10-17T14:49:14Z)
• LoopStack: a Lightweight Tensor Algebra Compiler Stack [61.04098601022665]
  LoopStack is a domain specific compiler stack for tensor operations. It generates machine code that matches and frequently exceeds the performance of in state-of-the-art machine learning frameworks. It has a very small memory footprint - a binary size of 245KB, and under 30K lines of effective code makes it ideal for use on mobile and embedded devices.
  arXiv  Detail & Related papers  (2022-05-02T01:57:58Z)
• Content-Aware Convolutional Neural Networks [98.97634685964819]
  Convolutional Neural Networks (CNNs) have achieved great success due to the powerful feature learning ability of convolution layers. We propose a Content-aware Convolution (CAC) that automatically detects the smooth windows and applies a 1x1 convolutional kernel to replace the original large kernel.
  arXiv  Detail & Related papers  (2021-06-30T03:54:35Z)
• Efficient Integer-Arithmetic-Only Convolutional Neural Networks [87.01739569518513]
  We replace conventional ReLU with Bounded ReLU and find that the decline is due to activation quantization. Our integer networks achieve equivalent performance as the corresponding FPN networks, but have only 1/4 memory cost and run 2x faster on modern GPU.
  arXiv  Detail & Related papers  (2020-06-21T08:23:03Z)
• Pyramidal Convolution: Rethinking Convolutional Neural Networks for Visual Recognition [98.10703825716142]
  This work introduces pyramidal convolution (PyConv), which is capable of processing the input at multiple filter scales. We present different architectures based on PyConv for four main tasks on visual recognition: image classification, video action classification/recognition, object detection and semantic image segmentation/parsing.
  arXiv  Detail & Related papers  (2020-06-20T10:19:29Z)
• PolyDL: Polyhedral Optimizations for Creation of High Performance DL primitives [55.79741270235602]
  We present compiler algorithms to automatically generate high performance implementations of Deep Learning primitives. We develop novel data reuse analysis algorithms using the polyhedral model. We also show that such a hybrid compiler plus a minimal library-use approach results in state-of-the-art performance.
  arXiv  Detail & Related papers  (2020-06-02T06:44:09Z)
• Performance Aware Convolutional Neural Network Channel Pruning for Embedded GPUs [6.035819238203187]
  We show that a reduction in the number of convolutional channels, pruning 12% of the initial size, is in some cases detrimental to performance. We also find examples where performance-aware pruning achieves the intended results, with performance speedups of 3x with cuDNN and above 10x with Arm Compute Library and TVM.
  arXiv  Detail & Related papers  (2020-02-20T12:07:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.