Using Python for Model Inference in Deep Learning

• URL: http://arxiv.org/abs/2104.00254v1
• Date: Thu, 1 Apr 2021 04:48:52 GMT
• Title: Using Python for Model Inference in Deep Learning
• Authors: Zachary DeVito, Jason Ansel, Will Constable, Michael Suo, Ailing Zhang, Kim Hazelwood
• Abstract summary: We show how it is possible to meet performance and packaging constraints while performing inference in Python. We present a way of using multiple Python interpreters within a single process to achieve scalable inference.
• Score: 0.6027358520885614
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Python has become the de-facto language for training deep neural networks, coupling a large suite of scientific computing libraries with efficient libraries for tensor computation such as PyTorch or TensorFlow. However, when models are used for inference they are typically extracted from Python as TensorFlow graphs or TorchScript programs in order to meet performance and packaging constraints. The extraction process can be time consuming, impeding fast prototyping. We show how it is possible to meet these performance and packaging constraints while performing inference in Python. In particular, we present a way of using multiple Python interpreters within a single process to achieve scalable inference and describe a new container format for models that contains both native Python code and data. This approach simplifies the model deployment story by eliminating the model extraction step, and makes it easier to integrate existing performance-enhancing Python libraries. We evaluate our design on a suite of popular PyTorch models on Github, showing how they can be packaged in our inference format, and comparing their performance to TorchScript. For larger models, our packaged Python models perform the same as TorchScript, and for smaller models where there is some Python overhead, our multi-interpreter approach ensures inference is still scalable.

Related papers

• PoTo: A Hybrid Andersen's Points-to Analysis for Python [3.6793233203143743]
  PoTo is an Andersen-style context-insensitive and flow-insensitive points-to analysis for Python. PoTo+ is a static type inference for Python built on the points-to analysis.
  arXiv  Detail & Related papers  (2024-09-05T21:26:25Z)
• PyMarian: Fast Neural Machine Translation and Evaluation in Python [11.291502854418098]
  We describe a Python interface to Marian NMT, a C++-based training and inference toolkit for sequence-to-sequence models. This interface enables models trained with Marian to be connected to the rich, wide range of tools available in Python.
  arXiv  Detail & Related papers  (2024-08-15T01:41:21Z)
• CyNetDiff -- A Python Library for Accelerated Implementation of Network Diffusion Models [0.9831489366502302]
  CyNetDiff is a Python library with components written in Cython to provide improved performance for these computationally intensive diffusion tasks. In many research tasks, these simulations are the most computationally intensive task, so it would be desirable to have a library for these with an interface to a high-level language.
  arXiv  Detail & Related papers  (2024-04-25T21:59:55Z)
• pyvene: A Library for Understanding and Improving PyTorch Models via Interventions [79.72930339711478]
  $textbfpyvene$ is an open-source library that supports customizable interventions on a range of different PyTorch modules. We show how $textbfpyvene$ provides a unified framework for performing interventions on neural models and sharing the intervened upon models with others.
  arXiv  Detail & Related papers  (2024-03-12T16:46:54Z)
• PyTorch Hyperparameter Tuning - A Tutorial for spotPython [0.20305676256390928]
  This tutorial includes a brief comparison with Ray Tune, a Python library for running experiments and tuning hyperparameters. We show that spotPython achieves similar or even better results while being more flexible and transparent than Ray Tune.
  arXiv  Detail & Related papers  (2023-05-19T17:47:50Z)
• Comparing neural network training performance between Elixir and Python [0.9023847175654603]
  Python has made a name for itself as one of the main programming languages. In February 2021, Jos'e Valim and Sean Moriarity published the first version of the Numerical Elixir (Nx) library.
  arXiv  Detail & Related papers  (2022-10-25T11:57:14Z)
• DADApy: Distance-based Analysis of DAta-manifolds in Python [51.37841707191944]
  DADApy is a python software package for analysing and characterising high-dimensional data. It provides methods for estimating the intrinsic dimension and the probability density, for performing density-based clustering and for comparing different distance metrics.
  arXiv  Detail & Related papers  (2022-05-04T08:41:59Z)
• PyGOD: A Python Library for Graph Outlier Detection [56.33769221859135]
  PyGOD is an open-source library for detecting outliers in graph data. It supports a wide array of leading graph-based methods for outlier detection. PyGOD is released under a BSD 2-Clause license at https://pygod.org and at the Python Package Index (PyPI)
  arXiv  Detail & Related papers  (2022-04-26T06:15:21Z)
• Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch [97.03321382630975]
  Continual Inference is a Python library for implementing Continual Inference Networks (CINs) in PyTorch. We offer a comprehensive introduction to CINs and their implementation in practice, and provide best-practices and code examples for composing complex modules for modern Deep Learning.
  arXiv  Detail & Related papers  (2022-04-07T13:03:09Z)
• PyHHMM: A Python Library for Heterogeneous Hidden Markov Models [63.01207205641885]
  PyHHMM is an object-oriented Python implementation of Heterogeneous-Hidden Markov Models (HHMMs) PyHHMM emphasizes features not supported in similar available frameworks: a heterogeneous observation model, missing data inference, different model order selection criterias, and semi-supervised training. PyHHMM relies on the numpy, scipy, scikit-learn, and seaborn Python packages, and is distributed under the Apache-2.0 License.
  arXiv  Detail & Related papers  (2022-01-12T07:32:36Z)
• OPFython: A Python-Inspired Optimum-Path Forest Classifier [68.8204255655161]
  This paper proposes a Python-based Optimum-Path Forest framework, denoted as OPFython. As OPFython is a Python-based library, it provides a more friendly environment and a faster prototyping workspace than the C language.
  arXiv  Detail & Related papers  (2020-01-28T15:46:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.