Comparative Analysis of Polynomial and Rational Approximations of
Hyperbolic Tangent Function for VLSI Implementation
- URL: http://arxiv.org/abs/2007.11976v2
- Date: Thu, 24 Sep 2020 13:41:18 GMT
- Title: Comparative Analysis of Polynomial and Rational Approximations of
Hyperbolic Tangent Function for VLSI Implementation
- Authors: Mahesh Chandra
- Abstract summary: Deep neural networks yield the state-of-the-art results in many computer vision and human machine interface applications such as object detection, speech recognition etc.
Since, these networks are computationally expensive, customized accelerators are designed for achieving the required performance at lower cost and power.
- Score: 5.429955391775968
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Deep neural networks yield the state-of-the-art results in many computer
vision and human machine interface applications such as object detection,
speech recognition etc. Since, these networks are computationally expensive,
customized accelerators are designed for achieving the required performance at
lower cost and power. One of the key building blocks of these neural networks
is non-linear activation function such as sigmoid, hyperbolic tangent (tanh),
and ReLU. A low complexity accurate hardware implementation of the activation
function is required to meet the performance and area targets of the neural
network accelerators. Even though, various methods and implementations of tanh
activation function have been published, a comparative study is missing. This
paper presents comparative analysis of polynomial and rational methods and
their hardware implementation.
Related papers
- The Promise of Analog Deep Learning: Recent Advances, Challenges and Opportunities [2.9312156642007303]
We evaluate and specify the advantages and disadvantages, along with the current progress with regards to deep learning, for analog implementations.
We identify the neural network-based experiments implemented using these hardware devices and discuss comparative performance achieved by the different analog deep learning methods.
Overall, we find that Analog Deep Learning has great potential for future consumer-level applications, but there is still a long road ahead in terms of scalability.
arXiv Detail & Related papers (2024-06-13T07:52:33Z) - Globally Optimal Training of Neural Networks with Threshold Activation
Functions [63.03759813952481]
We study weight decay regularized training problems of deep neural networks with threshold activations.
We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
arXiv Detail & Related papers (2023-03-06T18:59:13Z) - Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency.
We present an IPU-optimized release of our custom SNN Python package, snnTorch.
arXiv Detail & Related papers (2022-11-19T15:44:08Z) - Inducing Gaussian Process Networks [80.40892394020797]
We propose inducing Gaussian process networks (IGN), a simple framework for simultaneously learning the feature space as well as the inducing points.
The inducing points, in particular, are learned directly in the feature space, enabling a seamless representation of complex structured domains.
We report on experimental results for real-world data sets showing that IGNs provide significant advances over state-of-the-art methods.
arXiv Detail & Related papers (2022-04-21T05:27:09Z) - From DNNs to GANs: Review of efficient hardware architectures for deep
learning [0.0]
Neural network and deep learning has been started to impact the present research paradigm.
DSP processors are incapable of performing neural network, activation function, convolutional neural network and generative adversarial network operations.
Different algorithms have been adapted to design a DSP processor compatible for fast performance in neural network, activation function, convolutional neural network and generative adversarial network.
arXiv Detail & Related papers (2021-06-06T13:23:06Z) - Learning Frequency-aware Dynamic Network for Efficient Super-Resolution [56.98668484450857]
This paper explores a novel frequency-aware dynamic network for dividing the input into multiple parts according to its coefficients in the discrete cosine transform (DCT) domain.
In practice, the high-frequency part will be processed using expensive operations and the lower-frequency part is assigned with cheap operations to relieve the computation burden.
Experiments conducted on benchmark SISR models and datasets show that the frequency-aware dynamic network can be employed for various SISR neural architectures.
arXiv Detail & Related papers (2021-03-15T12:54:26Z) - Ps and Qs: Quantization-aware pruning for efficient low latency neural
network inference [56.24109486973292]
We study the interplay between pruning and quantization during the training of neural networks for ultra low latency applications.
We find that quantization-aware pruning yields more computationally efficient models than either pruning or quantization alone for our task.
arXiv Detail & Related papers (2021-02-22T19:00:05Z) - Activation function impact on Sparse Neural Networks [0.0]
Sparse Evolutionary Training allows for significantly lower computational complexity when compared to fully connected models.
This research provides insights into the relationship between the activation function used and the network performance at various sparsity levels.
arXiv Detail & Related papers (2020-10-12T18:05:04Z) - Hardware Implementation of Hyperbolic Tangent Function using Catmull-Rom
Spline Interpolation [5.429955391775968]
Deep neural networks yield the state of the art results in many computer vision and human machine interface tasks such as object recognition, speech recognition etc.
Since, these networks are computationally expensive, customized accelerators are designed for achieving the required performance at lower cost and power.
arXiv Detail & Related papers (2020-07-13T07:11:59Z) - Spiking Neural Networks Hardware Implementations and Challenges: a
Survey [53.429871539789445]
Spiking Neural Networks are cognitive algorithms mimicking neuron and synapse operational principles.
We present the state of the art of hardware implementations of spiking neural networks.
We discuss the strategies employed to leverage the characteristics of these event-driven algorithms at the hardware level.
arXiv Detail & Related papers (2020-05-04T13:24:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.