Performance-aware Approximation of Global Channel Pruning for Multitask CNNs

• URL: http://arxiv.org/abs/2303.11923v1
• Date: Tue, 21 Mar 2023 15:15:21 GMT
• Title: Performance-aware Approximation of Global Channel Pruning for Multitask CNNs
• Authors: Hancheng Ye, Bo Zhang, Tao Chen, Jiayuan Fan, and Bin Wang
• Abstract summary: Global channel pruning (GCP) aims to remove a subset of channels (filters) across different layers from a deep model without hurting the performance. We propose a Performance-Aware Global Channel Pruning (PAGCP) framework. Experiments on several multitask datasets show that the proposed PAGCP can reduce the FLOPs and parameters by over 60% with minor performance drop.
• Score: 13.356477450355547
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Global channel pruning (GCP) aims to remove a subset of channels (filters) across different layers from a deep model without hurting the performance. Previous works focus on either single task model pruning or simply adapting it to multitask scenario, and still face the following problems when handling multitask pruning: 1) Due to the task mismatch, a well-pruned backbone for classification task focuses on preserving filters that can extract category-sensitive information, causing filters that may be useful for other tasks to be pruned during the backbone pruning stage; 2) For multitask predictions, different filters within or between layers are more closely related and interacted than that for single task prediction, making multitask pruning more difficult. Therefore, aiming at multitask model compression, we propose a Performance-Aware Global Channel Pruning (PAGCP) framework. We first theoretically present the objective for achieving superior GCP, by considering the joint saliency of filters from intra- and inter-layers. Then a sequentially greedy pruning strategy is proposed to optimize the objective, where a performance-aware oracle criterion is developed to evaluate sensitivity of filters to each task and preserve the globally most task-related filters. Experiments on several multitask datasets show that the proposed PAGCP can reduce the FLOPs and parameters by over 60% with minor performance drop, and achieves 1.2x$\sim$3.3x acceleration on both cloud and mobile platforms.

Related papers

• Localizing Task Information for Improved Model Merging and Compression [61.16012721460561]
  We show that the information required to solve each task is still preserved after merging as different tasks mostly use non-overlapping sets of weights. We propose Consensus Merging, an algorithm that eliminates such weights and improves the general performance of existing model merging approaches.
  arXiv  Detail & Related papers  (2024-05-13T14:54:37Z)
• Mitigating Task Interference in Multi-Task Learning via Explicit Task Routing with Non-Learnable Primitives [19.90788777476128]
  Multi-task learning (MTL) seeks to learn a single model to accomplish multiple tasks by leveraging shared information among the tasks. Existing MTL models have been known to suffer from negative interference among tasks. We propose ETR-NLP to mitigate task interference through a synergistic combination of non-learnable primitives and explicit task routing.
  arXiv  Detail & Related papers  (2023-08-03T22:34:16Z)
• Structured Pruning for Multi-Task Deep Neural Networks [25.916166808223743]
  Multi-task deep neural network (DNN) models have computation and storage benefits over individual single-task models. We investigate the effectiveness of structured pruning on multi-task models.
  arXiv  Detail & Related papers  (2023-04-13T22:15:47Z)
• DiSparse: Disentangled Sparsification for Multitask Model Compression [92.84435347164435]
  DiSparse is a simple, effective, and first-of-its-kind multitask pruning and sparse training scheme. Our experimental results demonstrate superior performance on various configurations and settings.
  arXiv  Detail & Related papers  (2022-06-09T17:57:46Z)
• Task Adaptive Parameter Sharing for Multi-Task Learning [114.80350786535952]
  Adaptive Task Adapting Sharing (TAPS) is a method for tuning a base model to a new task by adaptively modifying a small, task-specific subset of layers. Compared to other methods, TAPS retains high accuracy on downstream tasks while introducing few task-specific parameters. We evaluate our method on a suite of fine-tuning tasks and architectures (ResNet, DenseNet, ViT) and show that it achieves state-of-the-art performance while being simple to implement.
  arXiv  Detail & Related papers  (2022-03-30T23:16:07Z)
• Multi-Task Classification of Sewer Pipe Defects and Properties using a Cross-Task Graph Neural Network Decoder [56.673599764041384]
  We present a novel decoder-focused multi-task classification architecture Cross-Task Graph Neural Network (CT-GNN) CT-GNN refines the disjointed per-task predictions using cross-task information. We achieve state-of-the-art performance on all four classification tasks in the Sewer-ML dataset.
  arXiv  Detail & Related papers  (2021-11-15T15:36:50Z)
• Conflict-Averse Gradient Descent for Multi-task Learning [56.379937772617]
  A major challenge in optimizing a multi-task model is the conflicting gradients. We introduce Conflict-Averse Gradient descent (CAGrad) which minimizes the average loss function. CAGrad balances the objectives automatically and still provably converges to a minimum over the average loss.
  arXiv  Detail & Related papers  (2021-10-26T22:03:51Z)
• Exploring Relational Context for Multi-Task Dense Prediction [76.86090370115]
  We consider a multi-task environment for dense prediction tasks, represented by a common backbone and independent task-specific heads. We explore various attention-based contexts, such as global and local, in the multi-task setting. We propose an Adaptive Task-Relational Context module, which samples the pool of all available contexts for each task pair.
  arXiv  Detail & Related papers  (2021-04-28T16:45:56Z)
• MTP: Multi-Task Pruning for Efficient Semantic Segmentation Networks [32.84644563020912]
  We present a multi-task channel pruning approach for semantic segmentation networks. The importance of each convolution filter wrt the channel of an arbitrary layer will be simultaneously determined by the classification and segmentation tasks. Experimental results on several benchmarks illustrate the superiority of the proposed algorithm over the state-of-the-art pruning methods.
  arXiv  Detail & Related papers  (2020-07-16T15:03:01Z)
• Conditional Channel Gated Networks for Task-Aware Continual Learning [44.894710899300435]
  Convolutional Neural Networks experience catastrophic forgetting when optimized on a sequence of learning problems. We introduce a novel framework to tackle this problem with conditional computation. We validate our proposal on four continual learning datasets.
  arXiv  Detail & Related papers  (2020-03-31T19:35:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.