Structured Pruning for Multi-Task Deep Neural Networks
- URL: http://arxiv.org/abs/2304.06840v1
- Date: Thu, 13 Apr 2023 22:15:47 GMT
- Title: Structured Pruning for Multi-Task Deep Neural Networks
- Authors: Siddhant Garg, Lijun Zhang, Hui Guan
- Abstract summary: Multi-task deep neural network (DNN) models have computation and storage benefits over individual single-task models.
We investigate the effectiveness of structured pruning on multi-task models.
- Score: 25.916166808223743
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Although multi-task deep neural network (DNN) models have computation and
storage benefits over individual single-task DNN models, they can be further
optimized via model compression. Numerous structured pruning methods are
already developed that can readily achieve speedups in single-task models, but
the pruning of multi-task networks has not yet been extensively studied. In
this work, we investigate the effectiveness of structured pruning on multi-task
models. We use an existing single-task filter pruning criterion and also
introduce an MTL-based filter pruning criterion for estimating the filter
importance scores. We prune the model using an iterative pruning strategy with
both pruning methods. We show that, with careful hyper-parameter tuning,
architectures obtained from different pruning methods do not have significant
differences in their performances across tasks when the number of parameters is
similar. We also show that iterative structure pruning may not be the best way
to achieve a well-performing pruned model because, at extreme pruning levels,
there is a high drop in performance across all tasks. But when the same models
are randomly initialized and re-trained, they show better results.
Related papers
- One-Shot Pruning for Fast-adapting Pre-trained Models on Devices [28.696989086706186]
Large-scale pre-trained models have been remarkably successful in resolving downstream tasks.
deploying these models on low-capability devices still requires an effective approach, such as model pruning.
We present a scalable one-shot pruning method that leverages pruned knowledge of similar tasks to extract a sub-network from the pre-trained model for a new task.
arXiv Detail & Related papers (2023-07-10T06:44:47Z) - DepGraph: Towards Any Structural Pruning [68.40343338847664]
We study general structural pruning of arbitrary architecture like CNNs, RNNs, GNNs and Transformers.
We propose a general and fully automatic method, emphDependency Graph (DepGraph), to explicitly model the dependency between layers and comprehensively group parameters for pruning.
In this work, we extensively evaluate our method on several architectures and tasks, including ResNe(X)t, DenseNet, MobileNet and Vision transformer for images, GAT for graph, DGCNN for 3D point cloud, alongside LSTM for language, and demonstrate that, even with a
arXiv Detail & Related papers (2023-01-30T14:02:33Z) - Voting from Nearest Tasks: Meta-Vote Pruning of Pre-trained Models for
Downstream Tasks [55.431048995662714]
We create a small model for a new task from the pruned models of similar tasks.
We show that a few fine-tuning steps on this model suffice to produce a promising pruned-model for the new task.
We develop a simple but effective ''Meta-Vote Pruning (MVP)'' method that significantly reduces the pruning iterations for a new task.
arXiv Detail & Related papers (2023-01-27T06:49:47Z) - Interpretations Steered Network Pruning via Amortized Inferred Saliency
Maps [85.49020931411825]
Convolutional Neural Networks (CNNs) compression is crucial to deploying these models in edge devices with limited resources.
We propose to address the channel pruning problem from a novel perspective by leveraging the interpretations of a model to steer the pruning process.
We tackle this challenge by introducing a selector model that predicts real-time smooth saliency masks for pruned models.
arXiv Detail & Related papers (2022-09-07T01:12:11Z) - DiSparse: Disentangled Sparsification for Multitask Model Compression [92.84435347164435]
DiSparse is a simple, effective, and first-of-its-kind multitask pruning and sparse training scheme.
Our experimental results demonstrate superior performance on various configurations and settings.
arXiv Detail & Related papers (2022-06-09T17:57:46Z) - Pruning Pretrained Encoders with a Multitask Objective [12.062758391661847]
We compare pruning a single model with a multitask objective against the best ensemble of single-task models.
Additional analysis finds that using a multitask objective during pruning can also be an effective method for reducing model sizes for low-resource tasks.
arXiv Detail & Related papers (2021-12-10T17:57:33Z) - One-Cycle Pruning: Pruning ConvNets Under a Tight Training Budget [0.0]
Introducing sparsity in a neural network has been an efficient way to reduce its complexity while keeping its performance almost intact.
Most of the time, sparsity is introduced using a three-stage pipeline: 1) train the model to convergence, 2) prune the model according to some criterion, 3) fine-tune the pruned model to recover performance.
In our work, we propose to get rid of the first step of the pipeline and to combine the two other steps in a single pruning-training cycle.
arXiv Detail & Related papers (2021-07-05T15:27:07Z) - Manifold Regularized Dynamic Network Pruning [102.24146031250034]
This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks.
The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
arXiv Detail & Related papers (2021-03-10T03:59:03Z) - DAIS: Automatic Channel Pruning via Differentiable Annealing Indicator
Search [55.164053971213576]
convolutional neural network has achieved great success in fulfilling computer vision tasks despite large computation overhead.
Structured (channel) pruning is usually applied to reduce the model redundancy while preserving the network structure.
Existing structured pruning methods require hand-crafted rules which may lead to tremendous pruning space.
arXiv Detail & Related papers (2020-11-04T07:43:01Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.