Nerva: a Truly Sparse Implementation of Neural Networks

• URL: http://arxiv.org/abs/2407.17437v1
• Date: Wed, 24 Jul 2024 17:13:31 GMT
• Title: Nerva: a Truly Sparse Implementation of Neural Networks
• Authors: Wieger Wesselink, Bram Grooten, Qiao Xiao, Cassio de Campos, Mykola Pechenizkiy,
• Abstract summary: Nerva is a fast neural network library under development in C++. It supports sparsity by using the sparse matrix operations of Intel's Math Kernel Library.
• Score: 16.29955529463831
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Nerva, a fast neural network library under development in C++. It supports sparsity by using the sparse matrix operations of Intel's Math Kernel Library (MKL), which eliminates the need for binary masks. We show that Nerva significantly decreases training time and memory usage while reaching equivalent accuracy to PyTorch. We run static sparse experiments with an MLP on CIFAR-10. On high sparsity levels like $99\%$, the runtime is reduced by a factor of $4\times$ compared to a PyTorch model using masks. Similar to other popular frameworks such as PyTorch and Keras, Nerva offers a Python interface for users to work with.

Related papers

• No Saved Kaleidosope: an 100% Jitted Neural Network Coding Language with Pythonic Syntax [0.8408735228878615]
  We developed a jitted compiler for training Artificial Neural Networks using C++, LLVM and Cuda. It features object-oriented characteristics, strong typing, parallel workers for data pre-processing, pythonic syntax for expressions, PyTorch like model declaration and Automatic Differentiation.
  arXiv  Detail & Related papers  (2024-09-17T23:15:39Z)
• KHNNs: hypercomplex neural networks computations via Keras using TensorFlow and PyTorch [0.0]
  We propose a library applications integrated with Keras that can do computations within Dense and PyTorch. It provides Dense and Convolutional 1D, 2D, and 3D layers architectures.
  arXiv  Detail & Related papers  (2024-06-29T14:36:37Z)
• iSpLib: A Library for Accelerating Graph Neural Networks using Auto-tuned Sparse Operations [1.3030767447016454]
  iSpLib is a PyTorch-based C++ library equipped with auto-tuned sparse operations. We demonstrate that iSpLib obtains up to 27x overall training speedup compared to the equivalent PyTorch 2.1.0 and PyTorch Geometric 2.4.0 implementations on the CPU.
  arXiv  Detail & Related papers  (2024-03-21T21:56:44Z)
• depyf: Open the Opaque Box of PyTorch Compiler for Machine Learning Researchers [92.13613958373628]
  textttdepyf is a tool designed to demystify the inner workings of the PyTorch compiler. textttdepyf decompiles bytecode generated by PyTorch back into equivalent source code.
  arXiv  Detail & Related papers  (2024-03-14T16:17:14Z)
• RoseNNa: A performant, portable library for neural network inference with application to computational fluid dynamics [0.0]
  We present the roseNNa library, which bridges the gap between neural network inference and CFD. RoseNNa is a non-invasive, lightweight (1000 lines) tool for neural network inference.
  arXiv  Detail & Related papers  (2023-07-30T21:11:55Z)
• torchgfn: A PyTorch GFlowNet library [56.071033896777784]
  torchgfn is a PyTorch library that aims to address this need. It provides users with a simple API for environments and useful abstractions for samplers and losses.
  arXiv  Detail & Related papers  (2023-05-24T00:20:59Z)
• Training Your Sparse Neural Network Better with Any Mask [106.134361318518]
  Pruning large neural networks to create high-quality, independently trainable sparse masks is desirable. In this paper we demonstrate an alternative opportunity: one can customize the sparse training techniques to deviate from the default dense network training protocols. Our new sparse training recipe is generally applicable to improving training from scratch with various sparse masks.
  arXiv  Detail & Related papers  (2022-06-26T00:37:33Z)
• Variable Bitrate Neural Fields [75.24672452527795]
  We present a dictionary method for compressing feature grids, reducing their memory consumption by up to 100x. We formulate the dictionary optimization as a vector-quantized auto-decoder problem which lets us learn end-to-end discrete neural representations in a space where no direct supervision is available.
  arXiv  Detail & Related papers  (2022-06-15T17:58:34Z)
• TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch Models [16.30276204466139]
  We introduce torchNTK, a python library to calculate the empirical neural tangent kernel (NTK) of neural network models in the PyTorch framework. A feature of the library is that we expose the user to layerwise NTK components, and show that in some regimes a layerwise calculation is more memory efficient.
  arXiv  Detail & Related papers  (2022-05-24T21:27:58Z)
• Hybrid Models for Learning to Branch [81.93868699246214]
  We propose a new hybrid architecture for efficient branching on CPU machines. The proposed architecture combines the expressive power of GNNs with computationally inexpensive multi-layer perceptrons (MLP) for branching.
  arXiv  Detail & Related papers  (2020-06-26T21:03:45Z)
• On the distance between two neural networks and the stability of learning [59.62047284234815]
  This paper relates parameter distance to gradient breakdown for a broad class of nonlinear compositional functions. The analysis leads to a new distance function called deep relative trust and a descent lemma for neural networks.
  arXiv  Detail & Related papers  (2020-02-09T19:18:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.