TaxoNN: A Light-Weight Accelerator for Deep Neural Network Training
- URL: http://arxiv.org/abs/2010.05197v1
- Date: Sun, 11 Oct 2020 09:04:19 GMT
- Title: TaxoNN: A Light-Weight Accelerator for Deep Neural Network Training
- Authors: Reza Hojabr, Kamyar Givaki, Kossar Pourahmadi, Parsa Nooralinejad,
Ahmad Khonsari, Dara Rahmati, M. Hassan Najafi
- Abstract summary: We present a novel approach to add the training ability to a baseline DNN accelerator (inference only) by splitting the SGD algorithm into simple computational elements.
Based on this approach we propose TaxoNN, a light-weight accelerator for DNN training.
Our experimental results show that TaxoNN delivers, on average, 0.97% higher misclassification rate compared to a full-precision implementation.
- Score: 2.5025363034899732
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Emerging intelligent embedded devices rely on Deep Neural Networks (DNNs) to
be able to interact with the real-world environment. This interaction comes
with the ability to retrain DNNs, since environmental conditions change
continuously in time. Stochastic Gradient Descent (SGD) is a widely used
algorithm to train DNNs by optimizing the parameters over the training data
iteratively. In this work, first we present a novel approach to add the
training ability to a baseline DNN accelerator (inference only) by splitting
the SGD algorithm into simple computational elements. Then, based on this
heuristic approach we propose TaxoNN, a light-weight accelerator for DNN
training. TaxoNN can easily tune the DNN weights by reusing the hardware
resources used in the inference process using a time-multiplexing approach and
low-bitwidth units. Our experimental results show that TaxoNN delivers, on
average, 0.97% higher misclassification rate compared to a full-precision
implementation. Moreover, TaxoNN provides 2.1$\times$ power saving and
1.65$\times$ area reduction over the state-of-the-art DNN training accelerator.
Related papers
- Quantum-Inspired Tensor Neural Networks for Option Pricing [4.3942901219301564]
Recent advances in deep learning have enabled us to address the curse of dimensionality (COD) by solving problems in higher dimensions.
A subset of such approaches of addressing the COD has led us to solving high-dimensional PDEs.
This has resulted in opening doors to solving a variety of real-world problems ranging from mathematical finance to control for industrial applications.
arXiv Detail & Related papers (2022-12-28T19:39:55Z) - Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency.
We present an IPU-optimized release of our custom SNN Python package, snnTorch.
arXiv Detail & Related papers (2022-11-19T15:44:08Z) - Training Spiking Neural Networks with Local Tandem Learning [96.32026780517097]
Spiking neural networks (SNNs) are shown to be more biologically plausible and energy efficient than their predecessors.
In this paper, we put forward a generalized learning rule, termed Local Tandem Learning (LTL)
We demonstrate rapid network convergence within five training epochs on the CIFAR-10 dataset while having low computational complexity.
arXiv Detail & Related papers (2022-10-10T10:05:00Z) - Training High-Performance Low-Latency Spiking Neural Networks by
Differentiation on Spike Representation [70.75043144299168]
Spiking Neural Network (SNN) is a promising energy-efficient AI model when implemented on neuromorphic hardware.
It is a challenge to efficiently train SNNs due to their non-differentiability.
We propose the Differentiation on Spike Representation (DSR) method, which could achieve high performance.
arXiv Detail & Related papers (2022-05-01T12:44:49Z) - Rethinking Pretraining as a Bridge from ANNs to SNNs [13.984523794353477]
Spiking neural networks (SNNs) are known as a typical kind of brain-inspired models with their unique features.
How to obtain a high-accuracy model has always been the main challenge in the field of SNN.
arXiv Detail & Related papers (2022-03-02T14:59:57Z) - Can Deep Neural Networks be Converted to Ultra Low-Latency Spiking
Neural Networks? [3.2108350580418166]
Spiking neural networks (SNNs) operate via binary spikes distributed over time.
SOTA training strategies for SNNs involve conversion from a non-spiking deep neural network (DNN)
We propose a new training algorithm that accurately captures these distributions, minimizing the error between the DNN and converted SNN.
arXiv Detail & Related papers (2021-12-22T18:47:45Z) - Sub-bit Neural Networks: Learning to Compress and Accelerate Binary
Neural Networks [72.81092567651395]
Sub-bit Neural Networks (SNNs) are a new type of binary quantization design tailored to compress and accelerate BNNs.
SNNs are trained with a kernel-aware optimization framework, which exploits binary quantization in the fine-grained convolutional kernel space.
Experiments on visual recognition benchmarks and the hardware deployment on FPGA validate the great potentials of SNNs.
arXiv Detail & Related papers (2021-10-18T11:30:29Z) - Shift-BNN: Highly-Efficient Probabilistic Bayesian Neural Network
Training via Memory-Friendly Pattern Retrieving [5.043640793217879]
We design and prototype the first highly efficient BNN training accelerator, named Shift-BNN, that is low-cost and scalable.
Shift-BNN achieves an average of 4.9x (up to 10.8x) boost in energy efficiency and 1.6x (up to 2.8x) speedup over the baseline DNN training accelerator.
arXiv Detail & Related papers (2021-10-07T15:20:53Z) - slimTrain -- A Stochastic Approximation Method for Training Separable
Deep Neural Networks [2.4373900721120285]
DeepTrain networks (DNNs) have shown their success as high-dimensional neural function approximators in many applications.
We propose slimTrain, a modest optimization method for training DNNs with reduced sensitivity to the choice hyper-dimensional datasets.
arXiv Detail & Related papers (2021-09-28T19:31:57Z) - Deep Time Delay Neural Network for Speech Enhancement with Full Data
Learning [60.20150317299749]
This paper proposes a deep time delay neural network (TDNN) for speech enhancement with full data learning.
To make full use of the training data, we propose a full data learning method for speech enhancement.
arXiv Detail & Related papers (2020-11-11T06:32:37Z) - Progressive Tandem Learning for Pattern Recognition with Deep Spiking
Neural Networks [80.15411508088522]
Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency.
We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
arXiv Detail & Related papers (2020-07-02T15:38:44Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.