Group channel pruning and spatial attention distilling for object
detection
- URL: http://arxiv.org/abs/2306.01526v1
- Date: Fri, 2 Jun 2023 13:26:23 GMT
- Title: Group channel pruning and spatial attention distilling for object
detection
- Authors: Yun Chu, Pu Li, Yong Bai, Zhuhua Hu, Yongqing Chen and Jiafeng Lu
- Abstract summary: We introduce a three-stage model compression method: dynamic sparse training, group channel pruning, and spatial attention distilling.
Our method reduces the parameters of the model by 64.7 % and the calculation by 34.9%.
- Score: 2.8675002818821542
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Due to the over-parameterization of neural networks, many model compression
methods based on pruning and quantization have emerged. They are remarkable in
reducing the size, parameter number, and computational complexity of the model.
However, most of the models compressed by such methods need the support of
special hardware and software, which increases the deployment cost. Moreover,
these methods are mainly used in classification tasks, and rarely directly used
in detection tasks. To address these issues, for the object detection network
we introduce a three-stage model compression method: dynamic sparse training,
group channel pruning, and spatial attention distilling. Firstly, to select out
the unimportant channels in the network and maintain a good balance between
sparsity and accuracy, we put forward a dynamic sparse training method, which
introduces a variable sparse rate, and the sparse rate will change with the
training process of the network. Secondly, to reduce the effect of pruning on
network accuracy, we propose a novel pruning method called group channel
pruning. In particular, we divide the network into multiple groups according to
the scales of the feature layer and the similarity of module structure in the
network, and then we use different pruning thresholds to prune the channels in
each group. Finally, to recover the accuracy of the pruned network, we use an
improved knowledge distillation method for the pruned network. Especially, we
extract spatial attention information from the feature maps of specific scales
in each group as knowledge for distillation. In the experiments, we use YOLOv4
as the object detection network and PASCAL VOC as the training dataset. Our
method reduces the parameters of the model by 64.7 % and the calculation by
34.9%.
Related papers
- Network Pruning via Feature Shift Minimization [8.593369249204132]
We propose a novel Feature Shift Minimization (FSM) method to compress CNN models, which evaluates the feature shift by converging the information of both features and filters.
The proposed method yields state-of-the-art performance on various benchmark networks and datasets, verified by extensive experiments.
arXiv Detail & Related papers (2022-07-06T12:50:26Z) - Group Fisher Pruning for Practical Network Compression [58.25776612812883]
We present a general channel pruning approach that can be applied to various complicated structures.
We derive a unified metric based on Fisher information to evaluate the importance of a single channel and coupled channels.
Our method can be used to prune any structures including those with coupled channels.
arXiv Detail & Related papers (2021-08-02T08:21:44Z) - Manifold Regularized Dynamic Network Pruning [102.24146031250034]
This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks.
The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
arXiv Detail & Related papers (2021-03-10T03:59:03Z) - ESPN: Extremely Sparse Pruned Networks [50.436905934791035]
We show that a simple iterative mask discovery method can achieve state-of-the-art compression of very deep networks.
Our algorithm represents a hybrid approach between single shot network pruning methods and Lottery-Ticket type approaches.
arXiv Detail & Related papers (2020-06-28T23:09:27Z) - ReMarNet: Conjoint Relation and Margin Learning for Small-Sample Image
Classification [49.87503122462432]
We introduce a novel neural network termed Relation-and-Margin learning Network (ReMarNet)
Our method assembles two networks of different backbones so as to learn the features that can perform excellently in both of the aforementioned two classification mechanisms.
Experiments on four image datasets demonstrate that our approach is effective in learning discriminative features from a small set of labeled samples.
arXiv Detail & Related papers (2020-06-27T13:50:20Z) - Ensembled sparse-input hierarchical networks for high-dimensional
datasets [8.629912408966145]
We show that dense neural networks can be a practical data analysis tool in settings with small sample sizes.
A proposed method appropriately prunes the network structure by tuning only two L1-penalty parameters.
On a collection of real-world datasets with different sizes, EASIER-net selected network architectures in a data-adaptive manner and achieved higher prediction accuracy than off-the-shelf methods on average.
arXiv Detail & Related papers (2020-05-11T02:08:53Z) - Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs.
Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2020-04-06T15:51:00Z) - A "Network Pruning Network" Approach to Deep Model Compression [62.68120664998911]
We present a filter pruning approach for deep model compression using a multitask network.
Our approach is based on learning a a pruner network to prune a pre-trained target network.
The compressed model produced by our approach is generic and does not need any special hardware/software support.
arXiv Detail & Related papers (2020-01-15T20:38:23Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.