IPPRO: Importance-based Pruning with PRojective Offset for Magnitude-indifferent Structural Pruning

• URL: http://arxiv.org/abs/2507.14171v2
• Date: Tue, 22 Jul 2025 05:37:08 GMT
• Title: IPPRO: Importance-based Pruning with PRojective Offset for Magnitude-indifferent Structural Pruning
• Authors: Jaeheun Jung, Jaehyuk Lee, Yeajin Lee, Donghun Lee,
• Abstract summary: The magnitude importance and many correlated modern importance criteria often limit the capacity of pruning decision.<n>We propose a novel pruning strategy to challenge this dominating effect of magnitude and provide fair chance to each filter to be pruned.<n>Our work debunks the size-matters'' myth in pruning and expands the frontier of importance-based pruning.
• Score: 3.5565647225207297
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: With the growth of demand on neural network compression methods, the structured pruning methods including importance-based approach are actively studied. The magnitude importance and many correlated modern importance criteria often limit the capacity of pruning decision, since the filters with larger magnitudes are not likely to be pruned if the smaller one didn't, even if it is redundant. In this paper, we propose a novel pruning strategy to challenge this dominating effect of magnitude and provide fair chance to each filter to be pruned, by placing it on projective space. After that, we observe the gradient descent movement whether the filters move toward the origin or not, to measure how the filter is likely to be pruned. This measurement is used to construct PROscore, a novel importance score for IPPRO, a novel importance-based structured pruning with magnitude-indifference. Our evaluation results shows that the proposed importance criteria using the projective space achieves near-lossless pruning by reducing the performance drop in pruning, with promising performance after the finetuning. Our work debunks the ``size-matters'' myth in pruning and expands the frontier of importance-based pruning both theoretically and empirically.

Related papers

• Catalyst: a Novel Regularizer for Structured Pruning with Auxiliary Extension of Parameter Space [3.2924107133234455]
  Structured pruning aims to reduce the size and computational cost of deep neural networks by removing entire filters or channels.<n>The proposed Catalyst regularization ensures fair pruning chance for each filters with theoretically provable zero bias to their magnitude.<n>Pruning results on various datasets and models are superior to state-of-the-art filter pruning methods.
  arXiv  Detail & Related papers  (2025-07-10T13:10:02Z)
• How Sparse Can We Prune A Deep Network: A Fundamental Limit Viewpoint [3.4396642896512977]
  Network pruning is a commonly used measure to alleviate the storage and computational burden of deep neural networks.<n>This work takes a first-principles approach, i.e. we'll impose the sparsity constraint on the loss function and leverage the framework of statistical dimension in convex geometry.<n>We identify two key factors that determine the pruning ratio limit, namely, weight magnitude and network sharpness.
  arXiv  Detail & Related papers  (2023-06-09T12:39:41Z)
• Toward domain generalized pruning by scoring out-of-distribution importance [19.26591372002258]
  Filter pruning has been widely used for compressing convolutional neural networks to reduce computation costs during the deployment stage. We conduct extensive empirical experiments and reveal that although the intra-domain performance could be maintained after filter pruning, the cross-domain performance will decay to a large extent. Experiments show that under the same pruning ratio, our method can achieve significantly better cross-domain generalization performance than the baseline filter pruning method.
  arXiv  Detail & Related papers  (2022-10-25T07:36:55Z)
• Asymptotic Soft Cluster Pruning for Deep Neural Networks [5.311178623385279]
  Filter pruning method introduces structural sparsity by removing selected filters. We propose a novel filter pruning method called Asymptotic Soft Cluster Pruning. Our method can achieve competitive results compared with many state-of-the-art algorithms.
  arXiv  Detail & Related papers  (2022-06-16T13:58:58Z)
• Neural Pruning via Growing Regularization [82.9322109208353]
  We extend regularization to tackle two central problems of pruning: pruning schedule and weight importance scoring. Specifically, we propose an L2 regularization variant with rising penalty factors and show it can bring significant accuracy gains. The proposed algorithms are easy to implement and scalable to large datasets and networks in both structured and unstructured pruning.
  arXiv  Detail & Related papers  (2020-12-16T20:16:28Z)
• Data Agnostic Filter Gating for Efficient Deep Networks [72.4615632234314]
  Current filter pruning methods mainly leverage feature maps to generate important scores for filters and prune those with smaller scores. In this paper, we propose a data filter pruning method that uses an auxiliary network named Dagger module to induce pruning. In addition, to help prune filters with certain FLOPs constraints, we leverage an explicit FLOPs-aware regularization to directly promote pruning filters toward target FLOPs.
  arXiv  Detail & Related papers  (2020-10-28T15:26:40Z)
• SCOP: Scientific Control for Reliable Neural Network Pruning [127.20073865874636]
  This paper proposes a reliable neural network pruning algorithm by setting up a scientific control. Redundant filters can be discovered in the adversarial process of different features. Our method can reduce 57.8% parameters and 60.2% FLOPs of ResNet-101 with only 0.01% top-1 accuracy loss on ImageNet.
  arXiv  Detail & Related papers  (2020-10-21T03:02:01Z)
• Layer-adaptive sparsity for the Magnitude-based Pruning [88.37510230946478]
  We propose a novel importance score for global pruning, coined layer-adaptive magnitude-based pruning (LAMP) score. LAMP consistently outperforms popular existing schemes for layerwise sparsity selection.
  arXiv  Detail & Related papers  (2020-10-15T09:14:02Z)
• To Filter Prune, or to Layer Prune, That Is The Question [13.450136532402226]
  We show the limitation of filter pruning methods in terms of latency reduction. We present a set of layer pruning methods based on different criteria that achieve higher latency reduction than filter pruning methods on similar accuracy. LayerPrune also outperforms handcrafted architectures such as Shufflenet, MobileNet, MNASNet and ResNet18 by 7.3%, 4.6%, 2.8% and 0.5% respectively on similar latency budget on ImageNet dataset.
  arXiv  Detail & Related papers  (2020-07-11T02:51:40Z)
• Exploring Weight Importance and Hessian Bias in Model Pruning [55.75546858514194]
  We provide a principled exploration of pruning by building on a natural notion of importance. For linear models, we show that this notion of importance is captured by scaling which connects to the well-known Hessian-based pruning algorithm. We identify settings in which weights become more important despite becoming smaller, which in turn leads to a catastrophic failure of magnitude-based pruning.
  arXiv  Detail & Related papers  (2020-06-19T00:15:55Z)
• Dependency Aware Filter Pruning [74.69495455411987]
  Pruning a proportion of unimportant filters is an efficient way to mitigate the inference cost. Previous work prunes filters according to their weight norms or the corresponding batch-norm scaling factors. We propose a novel mechanism to dynamically control the sparsity-inducing regularization so as to achieve the desired sparsity.
  arXiv  Detail & Related papers  (2020-05-06T07:41:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.