Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN

• URL: http://arxiv.org/abs/2107.14444v1
• Date: Fri, 30 Jul 2021 06:18:19 GMT
• Title: Manipulating Identical Filter Redundancy for Efficient Pruning on Deep and Complicated CNN
• Authors: Xiaohan Ding, Tianxiang Hao, Jungong Han, Yuchen Guo, Guiguang Ding
• Abstract summary: We propose a novel Centripetal SGD (C-SGD) to make some filters identical, resulting in ideal redundancy patterns. C-SGD delivers better performance because the redundancy is better organized, compared to the existing methods.
• Score: 126.88224745942456
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The existence of redundancy in Convolutional Neural Networks (CNNs) enables us to remove some filters/channels with acceptable performance drops. However, the training objective of CNNs usually tends to minimize an accuracy-related loss function without any attention paid to the redundancy, making the redundancy distribute randomly on all the filters, such that removing any of them may trigger information loss and accuracy drop, necessitating a following finetuning step for recovery. In this paper, we propose to manipulate the redundancy during training to facilitate network pruning. To this end, we propose a novel Centripetal SGD (C-SGD) to make some filters identical, resulting in ideal redundancy patterns, as such filters become purely redundant due to their duplicates; hence removing them does not harm the network. As shown on CIFAR and ImageNet, C-SGD delivers better performance because the redundancy is better organized, compared to the existing methods. The efficiency also characterizes C-SGD because it is as fast as regular SGD, requires no finetuning, and can be conducted simultaneously on all the layers even in very deep CNNs. Besides, C-SGD can improve the accuracy of CNNs by first training a model with the same architecture but wider layers then squeezing it into the original width.

Related papers

• CR-SFP: Learning Consistent Representation for Soft Filter Pruning [18.701621806529438]
  Soft filter pruning(SFP) has emerged as an effective pruning technique for allowing pruned filters to update and regrow to the network. We propose to mitigate this gap by learning consistent representation for soft filter pruning, dubbed as CR-SFP. CR-SFP is a simple yet effective training framework to improve the accuracy of P-NN without introducing any additional inference cost.
  arXiv  Detail & Related papers  (2023-12-17T06:41:04Z)
• Instant Complexity Reduction in CNNs using Locality-Sensitive Hashing [50.79602839359522]
  We propose HASTE (Hashing for Tractable Efficiency), a parameter-free and data-free module that acts as a plug-and-play replacement for any regular convolution module. We are able to drastically compress latent feature maps without sacrificing much accuracy by using locality-sensitive hashing (LSH) In particular, we are able to instantly drop 46.72% of FLOPs while only losing 1.25% accuracy by just swapping the convolution modules in a ResNet34 on CIFAR-10 for our HASTE module.
  arXiv  Detail & Related papers  (2023-09-29T13:09:40Z)
• Structured Network Pruning by Measuring Filter-wise Interactions [6.037167142826297]
  We propose a structured network pruning approach SNPFI (Structured Network Pruning by measuring Filter-wise Interaction) During the pruning, the SNPFI can automatically assign the proper sparsity based on the filter utilization strength. We empirically demonstrate the effectiveness of the SNPFI with several commonly used CNN models.
  arXiv  Detail & Related papers  (2023-07-03T05:26:05Z)
• Iterative Soft Shrinkage Learning for Efficient Image Super-Resolution [91.3781512926942]
  Image super-resolution (SR) has witnessed extensive neural network designs from CNN to transformer architectures. This work investigates the potential of network pruning for super-resolution iteration to take advantage of off-the-shelf network designs and reduce the underlying computational overhead. We propose a novel Iterative Soft Shrinkage-Percentage (ISS-P) method by optimizing the sparse structure of a randomly network at each and tweaking unimportant weights with a small amount proportional to the magnitude scale on-the-fly.
  arXiv  Detail & Related papers  (2023-03-16T21:06:13Z)
• A Passive Similarity based CNN Filter Pruning for Efficient Acoustic Scene Classification [23.661189257759535]
  We present a method to develop low-complexity convolutional neural networks (CNNs) for acoustic scene classification (ASC) We propose a passive filter pruning framework, where a few convolutional filters from the CNNs are eliminated to yield compressed CNNs. The proposed method is simple, reduces computations per inference by 27%, with 25% fewer parameters, with less than 1% drop in accuracy.
  arXiv  Detail & Related papers  (2022-03-29T17:00:06Z)
• Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs [69.3939291118954]
  Unstructured pruning is well suited to reduce the memory footprint of convolutional neural networks (CNNs) Standard unstructured pruning (SP) reduces the memory footprint of CNNs by setting filter elements to zero. We introduce interspace pruning (IP), a general tool to improve existing pruning methods.
  arXiv  Detail & Related papers  (2022-03-15T11:50:45Z)
• Tensor Reordering for CNN Compression [7.228285747845778]
  We show how parameter redundancy in Convolutional Neural Network (CNN) filters can be effectively reduced by pruning in spectral domain. Our approach is applied to pretrained CNNs and we show that minor additional fine-tuning allows our method to recover the original model performance.
  arXiv  Detail & Related papers  (2020-10-22T23:45:34Z)
• Holistic Filter Pruning for Efficient Deep Neural Networks [25.328005340524825]
  "Holistic Filter Pruning" (HFP) is a novel approach for common DNN training that is easy to implement and enables to specify accurate pruning rates. In various experiments, we give insights into the training and achieve state-of-the-art performance on CIFAR-10 and ImageNet.
  arXiv  Detail & Related papers  (2020-09-17T09:23:36Z)
• Filter Grafting for Deep Neural Networks [71.39169475500324]
  Filter grafting aims to improve the representation capability of Deep Neural Networks (DNNs) We develop an entropy-based criterion to measure the information of filters and an adaptive weighting strategy for balancing the grafted information among networks. For example, the grafted MobileNetV2 outperforms the non-grafted MobileNetV2 by about 7 percent on CIFAR-100 dataset.
  arXiv  Detail & Related papers  (2020-01-15T03:18:57Z)
• Discrimination-aware Network Pruning for Deep Model Compression [79.44318503847136]
  Existing pruning methods either train from scratch with sparsity constraints or minimize the reconstruction error between the feature maps of the pre-trained models and the compressed ones. We propose a simple-yet-effective method called discrimination-aware channel pruning (DCP) to choose the channels that actually contribute to the discriminative power. Experiments on both image classification and face recognition demonstrate the effectiveness of our methods.
  arXiv  Detail & Related papers  (2020-01-04T07:07:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.