Optimal channel selection with discrete QCQP

• URL: http://arxiv.org/abs/2202.12417v1
• Date: Thu, 24 Feb 2022 23:26:51 GMT
• Title: Optimal channel selection with discrete QCQP
• Authors: Yeonwoo Jeong, Deokjae Lee, Gaon An, Changyong Son, Hyun Oh Song
• Abstract summary: We propose a novel channel selection method that optimally selects channels via discrete QCQP. We also propose a quadratic model that accurately estimates the actual inference time of the pruned network. Our experiments on CIFAR-10 and ImageNet show our proposed pruning method outperforms other fixed-importance channel pruning methods on various network architectures.
• Score: 14.734454356396158
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reducing the high computational cost of large convolutional neural networks is crucial when deploying the networks to resource-constrained environments. We first show the greedy approach of recent channel pruning methods ignores the inherent quadratic coupling between channels in the neighboring layers and cannot safely remove inactive weights during the pruning procedure. Furthermore, due to these inactive weights, the greedy methods cannot guarantee to satisfy the given resource constraints and deviate with the true objective. In this regard, we propose a novel channel selection method that optimally selects channels via discrete QCQP, which provably prevents any inactive weights and guarantees to meet the resource constraints tightly in terms of FLOPs, memory usage, and network size. We also propose a quadratic model that accurately estimates the actual inference time of the pruned network, which allows us to adopt inference time as a resource constraint option. Furthermore, we generalize our method to extend the selection granularity beyond channels and handle non-sequential connections. Our experiments on CIFAR-10 and ImageNet show our proposed pruning method outperforms other fixed-importance channel pruning methods on various network architectures.

Related papers

• Soft Masking for Cost-Constrained Channel Pruning [17.138115344464513]
  Structured channel pruning has been shown to significantly accelerate inference time for convolution neural networks (CNNs) on modern hardware. Recent works permanently zero these channels during training, which we observe to significantly hamper final accuracy. We propose Soft Masking for cost-constrained Channel Pruning (SMCP) to allow pruned channels to adaptively return to the network.
  arXiv  Detail & Related papers  (2022-11-04T01:28:45Z)
• Searching for Network Width with Bilaterally Coupled Network [75.43658047510334]
  We introduce a new supernet called Bilaterally Coupled Network (BCNet) to address this issue. In BCNet, each channel is fairly trained and responsible for the same amount of network widths, thus each network width can be evaluated more accurately. We propose the first open-source width benchmark on macro structures named Channel-Bench-Macro for the better comparison of width search algorithms.
  arXiv  Detail & Related papers  (2022-03-25T15:32:46Z)
• CATRO: Channel Pruning via Class-Aware Trace Ratio Optimization [61.71504948770445]
  We propose a novel channel pruning method via Class-Aware Trace Ratio Optimization (CATRO) to reduce the computational burden and accelerate the model inference. We show that CATRO achieves higher accuracy with similar cost or lower cost with similar accuracy than other state-of-the-art channel pruning algorithms. Because of its class-aware property, CATRO is suitable to prune efficient networks adaptively for various classification subtasks, enhancing handy deployment and usage of deep networks in real-world applications.
  arXiv  Detail & Related papers  (2021-10-21T06:26:31Z)
• AdaPruner: Adaptive Channel Pruning and Effective Weights Inheritance [9.3421559369389]
  We propose a pruning framework that adaptively determines the number of each layer's channels as well as the wights inheritance criteria for sub-network. AdaPruner allows to obtain pruned network quickly, accurately and efficiently. On ImageNet, we reduce 32.8% FLOPs of MobileNetV2 with only 0.62% decrease for top-1 accuracy, which exceeds all previous state-of-the-art channel pruning methods.
  arXiv  Detail & Related papers  (2021-09-14T01:52:05Z)
• Channel Scaling: A Scale-and-Select Approach for Transfer Learning [2.6304695993930594]
  Transfer learning with pre-trained neural networks is a common strategy for training classifiers in medical image analysis. We propose a novel approach to efficiently build small and well performing networks by introducing the channel-scaling layers. By imposing L1 regularization and thresholding on the scaling weights, this framework iteratively removes unnecessary feature channels from a pre-trained model.
  arXiv  Detail & Related papers  (2021-03-22T23:26:57Z)
• End-to-end learnable EEG channel selection with deep neural networks [72.21556656008156]
  We propose a framework to embed the EEG channel selection in the neural network itself. We deal with the discrete nature of this new optimization problem by employing continuous relaxations of the discrete channel selection parameters. This generic approach is evaluated on two different EEG tasks.
  arXiv  Detail & Related papers  (2021-02-11T13:44:07Z)
• Operation-Aware Soft Channel Pruning using Differentiable Masks [51.04085547997066]
  We propose a data-driven algorithm, which compresses deep neural networks in a differentiable way by exploiting the characteristics of operations. We perform extensive experiments and achieve outstanding performance in terms of the accuracy of output networks.
  arXiv  Detail & Related papers  (2020-07-08T07:44:00Z)
• DMCP: Differentiable Markov Channel Pruning for Neural Networks [67.51334229530273]
  We propose a novel differentiable method for channel pruning, named Differentiable Markov Channel Pruning (DMCP) Our method is differentiable and can be directly optimized by gradient descent with respect to standard task loss and budget regularization. To validate the effectiveness of our method, we perform extensive experiments on Imagenet with ResNet and MobilenetV2.
  arXiv  Detail & Related papers  (2020-05-07T09:39:55Z)
• Discrimination-aware Network Pruning for Deep Model Compression [79.44318503847136]
  Existing pruning methods either train from scratch with sparsity constraints or minimize the reconstruction error between the feature maps of the pre-trained models and the compressed ones. We propose a simple-yet-effective method called discrimination-aware channel pruning (DCP) to choose the channels that actually contribute to the discriminative power. Experiments on both image classification and face recognition demonstrate the effectiveness of our methods.
  arXiv  Detail & Related papers  (2020-01-04T07:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.