MCTensor: A High-Precision Deep Learning Library with Multi-Component
Floating-Point
- URL: http://arxiv.org/abs/2207.08867v1
- Date: Mon, 18 Jul 2022 18:26:43 GMT
- Title: MCTensor: A High-Precision Deep Learning Library with Multi-Component
Floating-Point
- Authors: Tao Yu, Went Guo, Jianan Canal Li, Tiancheng Yuan, Christopher De Sa
- Abstract summary: We introduce MCTensor, a library based on PyTorch for providing general-purpose arithmetic for DL training.
Our algorithms achieve high precision and also benefits from heavily-optimized PyTorch floating-point arithmetic.
We evaluate MCTensor arithmetic against PyTorch native arithmetic for a series of tasks, where models using MCTensor in float16 would match or outperform the PyTorch model with float32 or float64 precision.
- Score: 30.563108553430354
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this paper, we introduce MCTensor, a library based on PyTorch for
providing general-purpose and high-precision arithmetic for DL training.
MCTensor is used in the same way as PyTorch Tensor: we implement multiple
basic, matrix-level computation operators and NN modules for MCTensor with
identical PyTorch interface. Our algorithms achieve high precision computation
and also benefits from heavily-optimized PyTorch floating-point arithmetic. We
evaluate MCTensor arithmetic against PyTorch native arithmetic for a series of
tasks, where models using MCTensor in float16 would match or outperform the
PyTorch model with float32 or float64 precision.
Related papers
- Nerva: a Truly Sparse Implementation of Neural Networks [16.29955529463831]
Nerva is a fast neural network library under development in C++.
It supports sparsity by using the sparse matrix operations of Intel's Math Kernel Library.
arXiv Detail & Related papers (2024-07-24T17:13:31Z) - iSpLib: A Library for Accelerating Graph Neural Networks using Auto-tuned Sparse Operations [1.3030767447016454]
iSpLib is a PyTorch-based C++ library equipped with auto-tuned sparse operations.
We demonstrate that iSpLib obtains up to 27x overall training speedup compared to the equivalent PyTorch 2.1.0 and PyTorch Geometric 2.4.0 implementations on the CPU.
arXiv Detail & Related papers (2024-03-21T21:56:44Z) - PyTorch Hyperparameter Tuning - A Tutorial for spotPython [0.20305676256390928]
This tutorial includes a brief comparison with Ray Tune, a Python library for running experiments and tuning hyperparameters.
We show that spotPython achieves similar or even better results while being more flexible and transparent than Ray Tune.
arXiv Detail & Related papers (2023-05-19T17:47:50Z) - PyHopper -- Hyperparameter optimization [51.40201315676902]
We present PyHopper, a black-box optimization platform for machine learning researchers.
PyHopper's goal is to integrate with existing code with minimal effort and run the optimization process with minimal necessary manual oversight.
With simplicity as the primary theme, PyHopper is powered by a single robust Markov-chain Monte-Carlo optimization algorithm.
arXiv Detail & Related papers (2022-10-10T14:35:01Z) - TorchNTK: A Library for Calculation of Neural Tangent Kernels of PyTorch
Models [16.30276204466139]
We introduce torchNTK, a python library to calculate the empirical neural tangent kernel (NTK) of neural network models in the PyTorch framework.
A feature of the library is that we expose the user to layerwise NTK components, and show that in some regimes a layerwise calculation is more memory efficient.
arXiv Detail & Related papers (2022-05-24T21:27:58Z) - PyGOD: A Python Library for Graph Outlier Detection [56.33769221859135]
PyGOD is an open-source library for detecting outliers in graph data.
It supports a wide array of leading graph-based methods for outlier detection.
PyGOD is released under a BSD 2-Clause license at https://pygod.org and at the Python Package Index (PyPI)
arXiv Detail & Related papers (2022-04-26T06:15:21Z) - PyHHMM: A Python Library for Heterogeneous Hidden Markov Models [63.01207205641885]
PyHHMM is an object-oriented Python implementation of Heterogeneous-Hidden Markov Models (HHMMs)
PyHHMM emphasizes features not supported in similar available frameworks: a heterogeneous observation model, missing data inference, different model order selection criterias, and semi-supervised training.
PyHHMM relies on the numpy, scipy, scikit-learn, and seaborn Python packages, and is distributed under the Apache-2.0 License.
arXiv Detail & Related papers (2022-01-12T07:32:36Z) - I-BERT: Integer-only BERT Quantization [78.43819756382103]
We propose I-BERT, a novel quantization scheme for Transformer based models.
I-BERT performs an end-to-end integer-only BERT inference without any floating point calculation.
We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to the full-precision baseline.
arXiv Detail & Related papers (2021-01-05T02:42:58Z) - MOGPTK: The Multi-Output Gaussian Process Toolkit [71.08576457371433]
We present MOGPTK, a Python package for multi-channel data modelling using Gaussian processes (GP)
The aim of this toolkit is to make multi-output GP (MOGP) models accessible to researchers, data scientists, and practitioners alike.
arXiv Detail & Related papers (2020-02-09T23:34:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.