Low-complexity Approximate Convolutional Neural Networks
- URL: http://arxiv.org/abs/2208.00087v1
- Date: Fri, 29 Jul 2022 21:59:29 GMT
- Title: Low-complexity Approximate Convolutional Neural Networks
- Authors: R. J. Cintra, S. Duffner, C. Garcia, A. Leite
- Abstract summary: We present an approach for minimizing the computational complexity of trained Convolutional Neural Networks (ConvNet)
The idea is to approximate all elements of a given ConvNet with efficient approximations capable of extreme reductions in computational complexity.
Such low-complexity structures pave the way for low-power, efficient hardware designs.
- Score: 1.7368964547487395
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this paper, we present an approach for minimizing the computational
complexity of trained Convolutional Neural Networks (ConvNet). The idea is to
approximate all elements of a given ConvNet and replace the original
convolutional filters and parameters (pooling and bias coefficients; and
activation function) with efficient approximations capable of extreme
reductions in computational complexity. Low-complexity convolution filters are
obtained through a binary (zero-one) linear programming scheme based on the
Frobenius norm over sets of dyadic rationals. The resulting matrices allow for
multiplication-free computations requiring only addition and bit-shifting
operations. Such low-complexity structures pave the way for low-power,
efficient hardware designs. We applied our approach on three use cases of
different complexity: (i) a "light" but efficient ConvNet for face detection
(with around 1000 parameters); (ii) another one for hand-written digit
classification (with more than 180000 parameters); and (iii) a significantly
larger ConvNet: AlexNet with $\approx$1.2 million matrices. We evaluated the
overall performance on the respective tasks for different levels of
approximations. In all considered applications, very low-complexity
approximations have been derived maintaining an almost equal classification
performance.
Related papers
- Transferable Neural Wavefunctions for Solids [5.219568203653524]
We show how to optimize a single ansatz across all of these variations.
We successfully transfer a network, pre-trained on 2x2x2 supercells of LiH, to 3x3x3 supercells.
arXiv Detail & Related papers (2024-05-13T09:59:59Z) - Self-concordant Smoothing for Large-Scale Convex Composite Optimization [0.0]
We introduce a notion of self-concordant smoothing for minimizing the sum of two convex functions, one of which is smooth and the other may be nonsmooth.
We prove the convergence of two resulting algorithms: Prox-N-SCORE, a proximal Newton algorithm and Prox-GGN-SCORE, a proximal generalized Gauss-Newton algorithm.
arXiv Detail & Related papers (2023-09-04T19:47:04Z) - A Generalization of Continuous Relaxation in Structured Pruning [0.3277163122167434]
Trends indicate that deeper and larger neural networks with an increasing number of parameters achieve higher accuracy than smaller neural networks.
We generalize structured pruning with algorithms for network augmentation, pruning, sub-network collapse and removal.
The resulting CNN executes efficiently on GPU hardware without computationally expensive sparse matrix operations.
arXiv Detail & Related papers (2023-08-28T14:19:13Z) - Efficient Convex Algorithms for Universal Kernel Learning [50.877957471649395]
An ideal set of kernels should: admit a linear parameterization (for tractability); dense in the set of all kernels (for accuracy)
Previous algorithms for optimization of kernels were limited to classification and relied on computationally complex Semidefinite Programming (SDP) algorithms.
We propose a SVD-QCQPQP algorithm which dramatically reduces the computational complexity as compared with previous SDP-based approaches.
arXiv Detail & Related papers (2023-04-15T04:57:37Z) - SVNet: Where SO(3) Equivariance Meets Binarization on Point Cloud
Representation [65.4396959244269]
The paper tackles the challenge by designing a general framework to construct 3D learning architectures.
The proposed approach can be applied to general backbones like PointNet and DGCNN.
Experiments on ModelNet40, ShapeNet, and the real-world dataset ScanObjectNN, demonstrated that the method achieves a great trade-off between efficiency, rotation, and accuracy.
arXiv Detail & Related papers (2022-09-13T12:12:19Z) - Optimization-based Block Coordinate Gradient Coding for Mitigating
Partial Stragglers in Distributed Learning [58.91954425047425]
This paper aims to design a new gradient coding scheme for mitigating partial stragglers in distributed learning.
We propose a gradient coordinate coding scheme with L coding parameters representing L possibly different diversities for the L coordinates, which generates most gradient coding schemes.
arXiv Detail & Related papers (2022-06-06T09:25:40Z) - Gaussian Elimination versus Greedy Methods for the Synthesis of Linear
Reversible Circuits [0.0]
reversible circuits represent a subclass of reversible circuits with many applications in quantum computing.
We propose a new algorithm for any linear reversible operator by using an optimized version of the Gaussian elimination algorithm and a tuned LU factorization.
Overall, our algorithms improve the state-of-the-art methods for specific ranges of problem sizes.
arXiv Detail & Related papers (2022-01-17T16:31:42Z) - Statistically Meaningful Approximation: a Case Study on Approximating
Turing Machines with Transformers [50.85524803885483]
This work proposes a formal definition of statistically meaningful (SM) approximation which requires the approximating network to exhibit good statistical learnability.
We study SM approximation for two function classes: circuits and Turing machines.
arXiv Detail & Related papers (2021-07-28T04:28:55Z) - Berrut Approximated Coded Computing: Straggler Resistance Beyond
Polynomial Computing [34.69732430310801]
We propose Berrut Approximated Coded Computing (BACC) as an alternative approach to deal with stragglers effect.
BACC is proven to be numerically stable with low computational complexity.
In particular, BACC is used to train a deep neural network on a cluster of servers.
arXiv Detail & Related papers (2020-09-17T14:23:38Z) - WrapNet: Neural Net Inference with Ultra-Low-Resolution Arithmetic [57.07483440807549]
We propose a method that adapts neural networks to use low-resolution (8-bit) additions in the accumulators, achieving classification accuracy comparable to their 32-bit counterparts.
We demonstrate the efficacy of our approach on both software and hardware platforms.
arXiv Detail & Related papers (2020-07-26T23:18:38Z) - Minimal Filtering Algorithms for Convolutional Neural Networks [82.24592140096622]
We develop fully parallel hardware-oriented algorithms for implementing the basic filtering operation for M=3,5,7,9, and 11.
A fully parallel hardware implementation of the proposed algorithms in each case gives approximately 30 percent savings in the number of embedded multipliers.
arXiv Detail & Related papers (2020-04-12T13:18:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.