Python Workflows on HPC Systems

• URL: http://arxiv.org/abs/2012.00365v1
• Date: Tue, 1 Dec 2020 09:51:12 GMT
• Title: Python Workflows on HPC Systems
• Authors: Dominik Strassel, Philipp Reusch and Janis Keuper
• Abstract summary: The recent successes and wide spread application of compute intensive machine learning and data analytics methods have been boosting the usage of the Python programming language on HPC systems. While Python provides many advantages for the users, it has not been designed with a focus on multi-user environments or parallel programming. In this paper, we analyze the key problems induced by the usage of Python on HPC clusters and sketch appropriate workarounds.
• Score: 2.1485350418225244
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The recent successes and wide spread application of compute intensive machine learning and data analytics methods have been boosting the usage of the Python programming language on HPC systems. While Python provides many advantages for the users, it has not been designed with a focus on multi-user environments or parallel programming - making it quite challenging to maintain stable and secure Python workflows on a HPC system. In this paper, we analyze the key problems induced by the usage of Python on HPC clusters and sketch appropriate workarounds for efficiently maintaining multi-user Python software environments, securing and restricting resources of Python jobs and containing Python processes, while focusing on Deep Learning applications running on GPU clusters.

Related papers

• PoTo: A Hybrid Andersen's Points-to Analysis for Python [3.6793233203143743]
  PoTo is an Andersen-style context-insensitive and flow-insensitive points-to analysis for Python. PoTo+ is a static type inference for Python built on the points-to analysis.
  arXiv  Detail & Related papers  (2024-09-05T21:26:25Z)
• A Comprehensive Guide to Combining R and Python code for Data Science, Machine Learning and Reinforcement Learning [42.350737545269105]
  We show how to run Python's scikit-learn, pytorch and OpenAI gym libraries for building Machine Learning, Deep Learning, and Reinforcement Learning projects easily.
  arXiv  Detail & Related papers  (2024-07-19T23:01:48Z)
• Python is Not Always the Best Choice: Embracing Multilingual Program of Thoughts [51.49688654641581]
  We propose a task and model agnostic approach called MultiPoT, which harnesses strength and diversity from various languages. Experimental results reveal that it significantly outperforms Python Self-Consistency. In particular, MultiPoT achieves more than 4.6% improvement on average on ChatGPT (gpt-3.5-turbo-0701)
  arXiv  Detail & Related papers  (2024-02-16T13:48:06Z)
• PyPOTS: A Python Toolbox for Data Mining on Partially-Observed Time Series [0.0]
  PyPOTS is an open-source Python library dedicated to data mining and analysis on partially-observed time series. It provides easy access to diverse algorithms categorized into four tasks: imputation, classification, clustering, and forecasting.
  arXiv  Detail & Related papers  (2023-05-30T07:57:05Z)
• PyGOD: A Python Library for Graph Outlier Detection [56.33769221859135]
  PyGOD is an open-source library for detecting outliers in graph data. It supports a wide array of leading graph-based methods for outlier detection. PyGOD is released under a BSD 2-Clause license at https://pygod.org and at the Python Package Index (PyPI)
  arXiv  Detail & Related papers  (2022-04-26T06:15:21Z)
• Continual Inference: A Library for Efficient Online Inference with Deep Neural Networks in PyTorch [97.03321382630975]
  Continual Inference is a Python library for implementing Continual Inference Networks (CINs) in PyTorch. We offer a comprehensive introduction to CINs and their implementation in practice, and provide best-practices and code examples for composing complex modules for modern Deep Learning.
  arXiv  Detail & Related papers  (2022-04-07T13:03:09Z)
• pymdp: A Python library for active inference in discrete state spaces [52.85819390191516]
  pymdp is an open-source package for simulating active inference in Python. We provide the first open-source package for simulating active inference with POMDPs.
  arXiv  Detail & Related papers  (2022-01-11T12:18:44Z)
• OMB-Py: Python Micro-Benchmarks for Evaluating Performance of MPI Libraries on HPC Systems [1.066106854070245]
  OMB-Py is the first communication benchmark suite for parallel Python applications. OMB-Py consists of a variety of point-to-point and collective communication benchmark tests. We report up to 106x speedup on 224 CPU cores compared to sequential execution.
  arXiv  Detail & Related papers  (2021-10-20T16:59:14Z)
• Extending Python for Quantum-Classical Computing via Quantum Just-in-Time Compilation [78.8942067357231]
  Python is a popular programming language known for its flexibility, usability, readability, and focus on developer productivity. We present a language extension to Python that enables heterogeneous quantum-classical computing via a robust C++ infrastructure for quantum just-in-time compilation.
  arXiv  Detail & Related papers  (2021-05-10T21:11:21Z)
• Using Python for Model Inference in Deep Learning [0.6027358520885614]
  We show how it is possible to meet performance and packaging constraints while performing inference in Python. We present a way of using multiple Python interpreters within a single process to achieve scalable inference.
  arXiv  Detail & Related papers  (2021-04-01T04:48:52Z)
• OPFython: A Python-Inspired Optimum-Path Forest Classifier [68.8204255655161]
  This paper proposes a Python-based Optimum-Path Forest framework, denoted as OPFython. As OPFython is a Python-based library, it provides a more friendly environment and a faster prototyping workspace than the C language.
  arXiv  Detail & Related papers  (2020-01-28T15:46:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.