Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural Networks

• URL: http://arxiv.org/abs/2108.13728v1
• Date: Tue, 31 Aug 2021 10:17:36 GMT
• Title: Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural Networks
• Authors: Zhouyang Xie, Yan Fu, Shengzhao Tian, Junlin Zhou, Duanbing Chen
• Abstract summary: A highly efficient pruning method is proposed to significantly reduce the cost of pruning DCNN. Our method shows competitive pruning performance among the state-of-the-art retraining-based pruning methods.
• Score: 0.9712140341805068
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Channel pruning is a promising technique to compress the parameters of deep convolutional neural networks(DCNN) and to speed up the inference. This paper aims to address the long-standing inefficiency of channel pruning. Most channel pruning methods recover the prediction accuracy by re-training the pruned model from the remaining parameters or random initialization. This re-training process is heavily dependent on the sufficiency of computational resources, training data, and human interference(tuning the training strategy). In this paper, a highly efficient pruning method is proposed to significantly reduce the cost of pruning DCNN. The main contributions of our method include: 1) pruning compensation, a fast and data-efficient substitute of re-training to minimize the post-pruning reconstruction loss of features, 2) compensation-aware pruning(CaP), a novel pruning algorithm to remove redundant or less-weighted channels by minimizing the loss of information, and 3) binary structural search with step constraint to minimize human interference. On benchmarks including CIFAR-10/100 and ImageNet, our method shows competitive pruning performance among the state-of-the-art retraining-based pruning methods and, more importantly, reduces the processing time by 95% and data usage by 90%.

Related papers

• DRIVE: Dual Gradient-Based Rapid Iterative Pruning [2.209921757303168]
  Modern deep neural networks (DNNs) consist of millions of parameters, necessitating high-performance computing during training and inference. Traditional pruning methods that are applied post-training focus on streamlining inference, but there are recent efforts to leverage sparsity early on by pruning before training. We present Dual Gradient-Based Rapid Iterative Pruning (DRIVE), which leverages dense training for initial epochs to counteract the randomness inherent at the inception.
  arXiv  Detail & Related papers  (2024-04-01T20:44:28Z)
• Towards Memory- and Time-Efficient Backpropagation for Training Spiking Neural Networks [70.75043144299168]
  Spiking Neural Networks (SNNs) are promising energy-efficient models for neuromorphic computing. We propose the Spatial Learning Through Time (SLTT) method that can achieve high performance while greatly improving training efficiency. Our method achieves state-of-the-art accuracy on ImageNet, while the memory cost and training time are reduced by more than 70% and 50%, respectively, compared with BPTT.
  arXiv  Detail & Related papers  (2023-02-28T05:01:01Z)
• Learning a Consensus Sub-Network with Polarization Regularization and One Pass Training [3.2214522506924093]
  Pruning schemes create extra overhead either by iterative training and fine-tuning for static pruning or repeated computation of a dynamic pruning graph. We propose a new parameter pruning strategy for learning a lighter-weight sub-network that minimizes the energy cost while maintaining comparable performance to the fully parameterised network on given downstream tasks. Our results on CIFAR-10 and CIFAR-100 suggest that our scheme can remove 50% of connections in deep networks with less than 1% reduction in classification accuracy.
  arXiv  Detail & Related papers  (2023-02-17T09:37:17Z)
• Trainability Preserving Neural Structured Pruning [64.65659982877891]
  We present trainability preserving pruning (TPP), a regularization-based structured pruning method that can effectively maintain trainability during sparsification. TPP can compete with the ground-truth dynamical isometry recovery method on linear networks. It delivers encouraging performance in comparison to many top-performing filter pruning methods.
  arXiv  Detail & Related papers  (2022-07-25T21:15:47Z)
• CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization [61.71504948770445]
  We propose a novel channel pruning method via Class-Aware Trace Ratio Optimization (CATRO) to reduce the computational burden and accelerate the model inference. We show that CATRO achieves higher accuracy with similar cost or lower cost with similar accuracy than other state-of-the-art channel pruning algorithms. Because of its class-aware property, CATRO is suitable to prune efficient networks adaptively for various classification subtasks, enhancing handy deployment and usage of deep networks in real-world applications.
  arXiv  Detail & Related papers  (2021-10-21T06:26:31Z)
• Sparse Training via Boosting Pruning Plasticity with Neuroregeneration [79.78184026678659]
  We study the effect of pruning throughout training from the perspective of pruning plasticity. We design a novel gradual magnitude pruning (GMP) method, named gradual pruning with zero-cost neuroregeneration (GraNet) and its dynamic sparse training (DST) variant (GraNet-ST) Perhaps most impressively, the latter for the first time boosts the sparse-to-sparse training performance over various dense-to-sparse methods by a large margin with ResNet-50 on ImageNet.
  arXiv  Detail & Related papers  (2021-06-19T02:09:25Z)
• Hessian-Aware Pruning and Optimal Neural Implant [74.3282611517773]
  Pruning is an effective method to reduce the memory footprint and FLOPs associated with neural network models. We introduce a new Hessian Aware Pruning method coupled with a Neural Implant approach that uses second-order sensitivity as a metric for structured pruning.
  arXiv  Detail & Related papers  (2021-01-22T04:08:03Z)
• UCP: Uniform Channel Pruning for Deep Convolutional Neural Networks Compression and Acceleration [24.42067007684169]
  We propose a novel uniform channel pruning (UCP) method to prune deep CNN. The unimportant channels, including convolutional kernels related to them, are pruned directly. We verify our method on CIFAR-10, CIFAR-100 and ILSVRC-2012 for image classification.
  arXiv  Detail & Related papers  (2020-10-03T01:51:06Z)
• Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning [83.59005356327103]
  A common limitation of most existing pruning techniques is that they require pre-training of the network at least once before pruning. We propose STAMP, which task-adaptively prunes a network pretrained on a large reference dataset by generating a pruning mask on it as a function of the target dataset. We validate STAMP against recent advanced pruning methods on benchmark datasets.
  arXiv  Detail & Related papers  (2020-06-22T10:57:43Z)
• Pruning Filters while Training for Efficiently Optimizing Deep Learning Networks [6.269700080380206]
  Pruning techniques have been proposed that remove less significant weights in deep networks. We propose a dynamic pruning-while-training procedure, wherein we prune filters of a deep network during training itself. Results indicate that pruning while training yields a compressed network with almost no accuracy loss after pruning 50% of the filters.
  arXiv  Detail & Related papers  (2020-03-05T18:05:17Z)
• Gradual Channel Pruning while Training using Feature Relevance Scores for Convolutional Neural Networks [6.534515590778012]
  Pruning is one of the predominant approaches used for deep network compression. We present a simple-yet-effective gradual channel pruning while training methodology using a novel data-driven metric. We demonstrate the effectiveness of the proposed methodology on architectures such as VGG and ResNet.
  arXiv  Detail & Related papers  (2020-02-23T17:56:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.