Structured Convolutions for Efficient Neural Network Design
- URL: http://arxiv.org/abs/2008.02454v2
- Date: Sat, 31 Oct 2020 04:41:55 GMT
- Title: Structured Convolutions for Efficient Neural Network Design
- Authors: Yash Bhalgat, Yizhe Zhang, Jamie Lin, Fatih Porikli
- Abstract summary: We tackle model efficiency by exploiting redundancy in the textitimplicit structure of the building blocks of convolutional neural networks.
We show how this decomposition can be applied to 2D and 3D kernels as well as the fully-connected layers.
- Score: 65.36569572213027
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: In this work, we tackle model efficiency by exploiting redundancy in the
\textit{implicit structure} of the building blocks of convolutional neural
networks. We start our analysis by introducing a general definition of
Composite Kernel structures that enable the execution of convolution operations
in the form of efficient, scaled, sum-pooling components. As its special case,
we propose \textit{Structured Convolutions} and show that these allow
decomposition of the convolution operation into a sum-pooling operation
followed by a convolution with significantly lower complexity and fewer
weights. We show how this decomposition can be applied to 2D and 3D kernels as
well as the fully-connected layers. Furthermore, we present a Structural
Regularization loss that promotes neural network layers to leverage on this
desired structure in a way that, after training, they can be decomposed with
negligible performance loss. By applying our method to a wide range of CNN
architectures, we demonstrate "structured" versions of the ResNets that are up
to 2$\times$ smaller and a new Structured-MobileNetV2 that is more efficient
while staying within an accuracy loss of 1% on ImageNet and CIFAR-10 datasets.
We also show similar structured versions of EfficientNet on ImageNet and HRNet
architecture for semantic segmentation on the Cityscapes dataset. Our method
performs equally well or superior in terms of the complexity reduction in
comparison to the existing tensor decomposition and channel pruning methods.
Related papers
- A Generalization of Continuous Relaxation in Structured Pruning [0.3277163122167434]
Trends indicate that deeper and larger neural networks with an increasing number of parameters achieve higher accuracy than smaller neural networks.
We generalize structured pruning with algorithms for network augmentation, pruning, sub-network collapse and removal.
The resulting CNN executes efficiently on GPU hardware without computationally expensive sparse matrix operations.
arXiv Detail & Related papers (2023-08-28T14:19:13Z) - DepGraph: Towards Any Structural Pruning [68.40343338847664]
We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers.
We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning.
In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
arXiv Detail & Related papers (2023-01-30T14:02:33Z) - Pushing the Efficiency Limit Using Structured Sparse Convolutions [82.31130122200578]
We propose Structured Sparse Convolution (SSC), which leverages the inherent structure in images to reduce the parameters in the convolutional filter.
We show that SSC is a generalization of commonly used layers (depthwise, groupwise and pointwise convolution) in efficient architectures''
Architectures based on SSC achieve state-of-the-art performance compared to baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet classification benchmarks.
arXiv Detail & Related papers (2022-10-23T18:37:22Z) - Convolutional Neural Network Compression through Generalized Kronecker
Product Decomposition [2.4240083226965115]
We compress layers by generalizing the Kronecker Product Decomposition to apply to multidimensionals, leading to the Generalized Kronecker Product Decomposition(GKPD)
Our approach yields a plug-and-play module that can be used as a drop-in replacement for any convolutional layer.
arXiv Detail & Related papers (2021-09-29T20:45:08Z) - Efficient Micro-Structured Weight Unification and Pruning for Neural
Network Compression [56.83861738731913]
Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices.
Previous unstructured or structured weight pruning methods can hardly truly accelerate inference.
We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
arXiv Detail & Related papers (2021-06-15T17:22:59Z) - ACDC: Weight Sharing in Atom-Coefficient Decomposed Convolution [57.635467829558664]
We introduce a structural regularization across convolutional kernels in a CNN.
We show that CNNs now maintain performance with dramatic reduction in parameters and computations.
arXiv Detail & Related papers (2020-09-04T20:41:47Z) - Growing Efficient Deep Networks by Structured Continuous Sparsification [34.7523496790944]
We develop an approach to growing deep network architectures over the course of training.
Our method can start from a small, simple seed architecture and dynamically grow and prune both layers and filters.
We achieve $49.7%$ inference FLOPs and $47.4%$ training FLOPs savings compared to a baseline ResNet-50 on ImageNet.
arXiv Detail & Related papers (2020-07-30T10:03:47Z) - Binarizing MobileNet via Evolution-based Searching [66.94247681870125]
We propose a use of evolutionary search to facilitate the construction and training scheme when binarizing MobileNet.
Inspired by one-shot architecture search frameworks, we manipulate the idea of group convolution to design efficient 1-Bit Convolutional Neural Networks (CNNs)
Our objective is to come up with a tiny yet efficient binary neural architecture by exploring the best candidates of the group convolution.
arXiv Detail & Related papers (2020-05-13T13:25:51Z) - Knapsack Pruning with Inner Distillation [11.04321604965426]
We propose a novel pruning method that optimize the final accuracy of the pruned network.
We prune the network channels while maintaining the high-level structure of the network.
Our method leads to state-of-the-art pruning results on ImageNet, CIFAR-10 and CIFAR-100 using ResNet backbones.
arXiv Detail & Related papers (2020-02-19T16:04:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.