AutoPruning for Deep Neural Network with Dynamic Channel Masking
- URL: http://arxiv.org/abs/2010.12021v2
- Date: Mon, 2 Nov 2020 04:06:51 GMT
- Title: AutoPruning for Deep Neural Network with Dynamic Channel Masking
- Authors: Baopu Li, Yanwen Fan, Zhihong Pan, Gang Zhang
- Abstract summary: We propose a learning based auto pruning algorithm for deep neural network.
A two objectives' problem that aims for the the weights and the best channels for each layer is first formulated.
An alternative optimization approach is then proposed to derive the optimal channel numbers and weights simultaneously.
- Score: 28.018077874687343
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Modern deep neural network models are large and computationally intensive.
One typical solution to this issue is model pruning. However, most current
pruning algorithms depend on hand crafted rules or domain expertise. To
overcome this problem, we propose a learning based auto pruning algorithm for
deep neural network, which is inspired by recent automatic machine
learning(AutoML). A two objectives' problem that aims for the the weights and
the best channels for each layer is first formulated. An alternative
optimization approach is then proposed to derive the optimal channel numbers
and weights simultaneously. In the process of pruning, we utilize a searchable
hyperparameter, remaining ratio, to denote the number of channels in each
convolution layer, and then a dynamic masking process is proposed to describe
the corresponding channel evolution. To control the trade-off between the
accuracy of a model and the pruning ratio of floating point operations, a novel
loss function is further introduced. Preliminary experimental results on
benchmark datasets demonstrate that our scheme achieves competitive results for
neural network pruning.
Related papers
- Concurrent Training and Layer Pruning of Deep Neural Networks [0.0]
We propose an algorithm capable of identifying and eliminating irrelevant layers of a neural network during the early stages of training.
We employ a structure using residual connections around nonlinear network sections that allow the flow of information through the network once a nonlinear section is pruned.
arXiv Detail & Related papers (2024-06-06T23:19:57Z) - Learning to Learn with Generative Models of Neural Network Checkpoints [71.06722933442956]
We construct a dataset of neural network checkpoints and train a generative model on the parameters.
We find that our approach successfully generates parameters for a wide range of loss prompts.
We apply our method to different neural network architectures and tasks in supervised and reinforcement learning.
arXiv Detail & Related papers (2022-09-26T17:59:58Z) - Neuron-based Pruning of Deep Neural Networks with Better Generalization
using Kronecker Factored Curvature Approximation [18.224344440110862]
The proposed algorithm directs the parameters of the compressed model toward a flatter solution by exploring the spectral radius of Hessian.
Our result shows that it improves the state-of-the-art results on neuron compression.
The method is able to achieve very small networks with small accuracy across different neural network models.
arXiv Detail & Related papers (2021-11-16T15:55:59Z) - Exploring Gradient Flow Based Saliency for DNN Model Compression [21.993801817422572]
Model pruning aims to reduce the deep neural network (DNN) model size or computational overhead.
Traditional model pruning methods that evaluates the channel significance for DNN pay too much attention to the local analysis of each channel.
We propose a new model pruning method from a new perspective of flow in this paper.
arXiv Detail & Related papers (2021-10-24T16:09:40Z) - CONet: Channel Optimization for Convolutional Neural Networks [33.58529066005248]
We study channel size optimization in convolutional neural networks (CNN)
We introduce an efficient dynamic scaling algorithm -- CONet -- that automatically optimize channel sizes across network layers for a given CNN.
We conduct experiments on CIFAR10/100 and ImageNet datasets and show that CONet can find efficient and accurate architectures searched in ResNet, DARTS, and DARTS+ spaces.
arXiv Detail & Related papers (2021-08-15T21:48:25Z) - End-to-end learnable EEG channel selection with deep neural networks [72.21556656008156]
We propose a framework to embed the EEG channel selection in the neural network itself.
We deal with the discrete nature of this new optimization problem by employing continuous relaxations of the discrete channel selection parameters.
This generic approach is evaluated on two different EEG tasks.
arXiv Detail & Related papers (2021-02-11T13:44:07Z) - Communication-Efficient Distributed Stochastic AUC Maximization with
Deep Neural Networks [50.42141893913188]
We study a distributed variable for large-scale AUC for a neural network as with a deep neural network.
Our model requires a much less number of communication rounds and still a number of communication rounds in theory.
Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
arXiv Detail & Related papers (2020-05-05T18:08:23Z) - Network Adjustment: Channel Search Guided by FLOPs Utilization Ratio [101.84651388520584]
This paper presents a new framework named network adjustment, which considers network accuracy as a function of FLOPs.
Experiments on standard image classification datasets and a wide range of base networks demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2020-04-06T15:51:00Z) - DHP: Differentiable Meta Pruning via HyperNetworks [158.69345612783198]
This paper introduces a differentiable pruning method via hypernetworks for automatic network pruning.
Latent vectors control the output channels of the convolutional layers in the backbone network and act as a handle for the pruning of the layers.
Experiments are conducted on various networks for image classification, single image super-resolution, and denoising.
arXiv Detail & Related papers (2020-03-30T17:59:18Z) - Model Fusion via Optimal Transport [64.13185244219353]
We present a layer-wise model fusion algorithm for neural networks.
We show that this can successfully yield "one-shot" knowledge transfer between neural networks trained on heterogeneous non-i.i.d. data.
arXiv Detail & Related papers (2019-10-12T22:07:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.