Non-Parametric Adaptive Network Pruning

• URL: http://arxiv.org/abs/2101.07985v2
• Date: Mon, 25 Jan 2021 08:44:26 GMT
• Title: Non-Parametric Adaptive Network Pruning
• Authors: Mingbao Lin, Rongrong Ji, Shaojie Li, Yan Wang, Yongjian Wu, Feiyue Huang, Qixiang Ye
• Abstract summary: We introduce non-parametric modeling to simplify the algorithm design. Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars. EPruner breaks the dependency on the training data in determining the "important" filters.
• Score: 125.4414216272874
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Popular network pruning algorithms reduce redundant information by optimizing hand-crafted parametric models, and may cause suboptimal performance and long time in selecting filters. We innovatively introduce non-parametric modeling to simplify the algorithm design, resulting in an automatic and efficient pruning approach called EPruner. Inspired by the face recognition community, we use a message passing algorithm Affinity Propagation on the weight matrices to obtain an adaptive number of exemplars, which then act as the preserved filters. EPruner breaks the dependency on the training data in determining the "important" filters and allows the CPU implementation in seconds, an order of magnitude faster than GPU based SOTAs. Moreover, we show that the weights of exemplars provide a better initialization for the fine-tuning. On VGGNet-16, EPruner achieves a 76.34%-FLOPs reduction by removing 88.80% parameters, with 0.06% accuracy improvement on CIFAR-10. In ResNet-152, EPruner achieves a 65.12%-FLOPs reduction by removing 64.18% parameters, with only 0.71% top-5 accuracy loss on ILSVRC-2012. Code can be available at https://github.com/lmbxmu/EPruner.

Related papers

• Instant Complexity Reduction in CNNs using Locality-Sensitive Hashing [50.79602839359522]
  We propose HASTE (Hashing for Tractable Efficiency), a parameter-free and data-free module that acts as a plug-and-play replacement for any regular convolution module. We are able to drastically compress latent feature maps without sacrificing much accuracy by using locality-sensitive hashing (LSH) In particular, we are able to instantly drop 46.72% of FLOPs while only losing 1.25% accuracy by just swapping the convolution modules in a ResNet34 on CIFAR-10 for our HASTE module.
  arXiv  Detail & Related papers  (2023-09-29T13:09:40Z)
• Filter Pruning for Efficient CNNs via Knowledge-driven Differential Filter Sampler [103.97487121678276]
  Filter pruning simultaneously accelerates the computation and reduces the memory overhead of CNNs. We propose a novel Knowledge-driven Differential Filter Sampler(KDFS) with Masked Filter Modeling(MFM) framework for filter pruning.
  arXiv  Detail & Related papers  (2023-07-01T02:28:41Z)
• AUTOSPARSE: Towards Automated Sparse Training of Deep Neural Networks [2.6742343015805083]
  We propose Gradient Annealing (GA) to explore the non-uniform distribution of sparsity inherent within neural networks. GA provides an elegant trade-off between sparsity and accuracy without the need for additional sparsity-inducing regularization. We integrate GA with the latest learnable pruning methods to create an automated sparse training algorithm called AutoSparse.
  arXiv  Detail & Related papers  (2023-04-14T06:19:07Z)
• Quantized Neural Networks for Low-Precision Accumulation with Guaranteed Overflow Avoidance [68.8204255655161]
  We introduce a quantization-aware training algorithm that guarantees avoiding numerical overflow when reducing the precision of accumulators during inference. We evaluate our algorithm across multiple quantized models that we train for different tasks, showing that our approach can reduce the precision of accumulators while maintaining model accuracy with respect to a floating-point baseline.
  arXiv  Detail & Related papers  (2023-01-31T02:46:57Z)
• Input Normalized Stochastic Gradient Descent Training of Deep Neural Networks [2.1485350418225244]
  In this paper, we propose a novel optimization algorithm for training machine learning models called Input Normalized Gradient Descent (INSGD) Our algorithm updates the network weights using gradient descent with $ell_$ and $ell_$-based normalizations applied to the learning rate, similar to NLMS. We evaluate the efficiency of our training algorithm on benchmark datasets using ResNet-18, WResNet-20, ResNet-50, and a toy neural network.
  arXiv  Detail & Related papers  (2022-12-20T00:08:37Z)
• Pruning-aware Sparse Regularization for Network Pruning [38.13617051756663]
  Structural neural network pruning aims to remove the redundant channels in the deep convolutional neural networks (CNNs) In this paper, we analyze these sparsity-training-based methods and find that the regularization of unpruned channels is unnecessary. We propose a novel pruning method, named MaskSparsity, with pruning-aware sparse regularization.
  arXiv  Detail & Related papers  (2022-01-18T07:19:23Z)
• Automatic Neural Network Pruning that Efficiently Preserves the Model Accuracy [2.538209532048867]
  pruning filters is a common solution, but most existing pruning methods do not preserve the model accuracy efficiently. We propose an automatic pruning method that learns which neurons to preserve in order to maintain the model accuracy while reducing the FLOPs to a predefined target. We achieve a 52.00% FLOPs reduction on ResNet-50, with a Top-1 accuracy of 47.51% after pruning and a state-of-the-art (SOTA) accuracy of 76.63% after finetuning.
  arXiv  Detail & Related papers  (2021-11-18T11:29:35Z)
• Training Compact CNNs for Image Classification using Dynamic-coded Filter Fusion [139.71852076031962]
  We present a novel filter pruning method, dubbed dynamic-coded filter fusion (DCFF) We derive compact CNNs in a computation-economical and regularization-free manner for efficient image classification. Our DCFF derives a compact VGGNet-16 with only 72.77M FLOPs and 1.06M parameters while reaching top-1 accuracy of 93.47%.
  arXiv  Detail & Related papers  (2021-07-14T18:07:38Z)
• Filter Sketch for Network Pruning [184.41079868885265]
  We propose a novel network pruning approach by information preserving of pre-trained network weights (filters) Our approach, referred to as FilterSketch, encodes the second-order information of pre-trained weights. Experiments on CIFAR-10 show that FilterSketch reduces 63.3% of FLOPs and prunes 59.9% of network parameters with negligible accuracy cost.
  arXiv  Detail & Related papers  (2020-01-23T13:57:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.