A Greedy Hierarchical Approach to Whole-Network Filter-Pruning in CNNs

• URL: http://arxiv.org/abs/2409.03777v2
• Date: Mon, 9 Sep 2024 05:58:29 GMT
• Title: A Greedy Hierarchical Approach to Whole-Network Filter-Pruning in CNNs
• Authors: Kiran Purohit, Anurag Reddy Parvathgari, Sourangshu Bhattacharya,
• Abstract summary: Whole-network filter pruning algorithms prune varying fractions of filters from each layer, hence providing greater flexibility. This paper proposes a two-level hierarchical approach for whole-network filter pruning which is efficient and uses the classification loss as the final criterion. Our method reduces the RAM requirement for ResNext101 from 7.6 GB to 1.5 GB and achieves a 94% reduction in FLOPS without losing accuracy on CIFAR-10.
• Score: 2.188091591747149
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Deep convolutional neural networks (CNNs) have achieved impressive performance in many computer vision tasks. However, their large model sizes require heavy computational resources, making pruning redundant filters from existing pre-trained CNNs an essential task in developing efficient models for resource-constrained devices. Whole-network filter pruning algorithms prune varying fractions of filters from each layer, hence providing greater flexibility. Current whole-network pruning methods are either computationally expensive due to the need to calculate the loss for each pruned filter using a training dataset, or use various heuristic / learned criteria for determining the pruning fractions for each layer. This paper proposes a two-level hierarchical approach for whole-network filter pruning which is efficient and uses the classification loss as the final criterion. The lower-level algorithm (called filter-pruning) uses a sparse-approximation formulation based on linear approximation of filter weights. We explore two algorithms: orthogonal matching pursuit-based greedy selection and a greedy backward pruning approach. The backward pruning algorithm uses a novel closed-form error criterion for efficiently selecting the optimal filter at each stage, thus making the whole algorithm much faster. The higher-level algorithm (called layer-selection) greedily selects the best-pruned layer (pruning using the filter-selection algorithm) using a global pruning criterion. We propose algorithms for two different global-pruning criteria: (1) layer-wise relative error (HBGS), and (2) final classification error (HBGTS). Our suite of algorithms outperforms state-of-the-art pruning methods on ResNet18, ResNet32, ResNet56, VGG16, and ResNext101. Our method reduces the RAM requirement for ResNext101 from 7.6 GB to 1.5 GB and achieves a 94% reduction in FLOPS without losing accuracy on CIFAR-10.

Related papers

• Efficient CNNs via Passive Filter Pruning [23.661189257759535]
  Convolutional neural networks (CNNs) have shown state-of-the-art performance in various applications. CNNs are resource-hungry due to their requirement of high computational complexity and memory storage. Recent efforts toward achieving computational efficiency in CNNs involve filter pruning methods.
  arXiv  Detail & Related papers  (2023-04-05T09:19:19Z)
• Sub-network Multi-objective Evolutionary Algorithm for Filter Pruning [5.998027804346945]
  Filter pruning is a common method to achieve model compression and acceleration in deep neural networks (DNNs) We propose a Sub-network Multiobjective Evolutionary Algorithm (SMOEA) for filter pruning. Experiments on VGG-14 model for CIFAR-10 verify the effectiveness of the proposed SMOEA.
  arXiv  Detail & Related papers  (2022-10-22T13:34:14Z)
• End-to-End Sensitivity-Based Filter Pruning [49.61707925611295]
  We present a sensitivity-based filter pruning algorithm (SbF-Pruner) to learn the importance scores of filters of each layer end-to-end. Our method learns the scores from the filter weights, enabling it to account for the correlations between the filters of each layer.
  arXiv  Detail & Related papers  (2022-04-15T10:21:05Z)
• A Passive Similarity based CNN Filter Pruning for Efficient Acoustic Scene Classification [23.661189257759535]
  We present a method to develop low-complexity convolutional neural networks (CNNs) for acoustic scene classification (ASC) We propose a passive filter pruning framework, where a few convolutional filters from the CNNs are eliminated to yield compressed CNNs. The proposed method is simple, reduces computations per inference by 27%, with 25% fewer parameters, with less than 1% drop in accuracy.
  arXiv  Detail & Related papers  (2022-03-29T17:00:06Z)
• Improve Convolutional Neural Network Pruning by Maximizing Filter Variety [0.0]
  Neural network pruning is a widely used strategy for reducing model storage and computing requirements. Common pruning criteria, such as l1-norm or movement, usually do not consider the individual utility of filters. We present a technique solving those two issues, and which can be appended to any pruning criteria.
  arXiv  Detail & Related papers  (2022-03-11T09:00:59Z)
• Pruning Networks with Cross-Layer Ranking & k-Reciprocal Nearest Filters [151.2423480789271]
  A novel pruning method, termed CLR-RNF, is proposed for filter-level network pruning. We conduct image classification on CIFAR-10 and ImageNet to demonstrate the superiority of our CLR-RNF over the state-of-the-arts.
  arXiv  Detail & Related papers  (2022-02-15T04:53:24Z)
• Manifold Regularized Dynamic Network Pruning [102.24146031250034]
  This paper proposes a new paradigm that dynamically removes redundant filters by embedding the manifold information of all instances into the space of pruned networks. The effectiveness of the proposed method is verified on several benchmarks, which shows better performance in terms of both accuracy and computational cost.
  arXiv  Detail & Related papers  (2021-03-10T03:59:03Z)
• Towards Optimally Efficient Tree Search with Deep Learning [76.64632985696237]
  This paper investigates the classical integer least-squares problem which estimates signals integer from linear models. The problem is NP-hard and often arises in diverse applications such as signal processing, bioinformatics, communications and machine learning. We propose a general hyper-accelerated tree search (HATS) algorithm by employing a deep neural network to estimate the optimal estimation for the underlying simplified memory-bounded A* algorithm.
  arXiv  Detail & Related papers  (2021-01-07T08:00:02Z)
• Data Agnostic Filter Gating for Efficient Deep Networks [72.4615632234314]
  Current filter pruning methods mainly leverage feature maps to generate important scores for filters and prune those with smaller scores. In this paper, we propose a data filter pruning method that uses an auxiliary network named Dagger module to induce pruning. In addition, to help prune filters with certain FLOPs constraints, we leverage an explicit FLOPs-aware regularization to directly promote pruning filters toward target FLOPs.
  arXiv  Detail & Related papers  (2020-10-28T15:26:40Z)
• Dependency Aware Filter Pruning [74.69495455411987]
  Pruning a proportion of unimportant filters is an efficient way to mitigate the inference cost. Previous work prunes filters according to their weight norms or the corresponding batch-norm scaling factors. We propose a novel mechanism to dynamically control the sparsity-inducing regularization so as to achieve the desired sparsity.
  arXiv  Detail & Related papers  (2020-05-06T07:41:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.