ICE-Pruning: An Iterative Cost-Efficient Pruning Pipeline for Deep Neural Networks

• URL: http://arxiv.org/abs/2505.07411v2
• Date: Sun, 15 Jun 2025 08:13:14 GMT
• Title: ICE-Pruning: An Iterative Cost-Efficient Pruning Pipeline for Deep Neural Networks
• Authors: Wenhao Hu, Paul Henderson, José Cano,
• Abstract summary: ICE-Pruning is an iterative pruning pipeline for Deep Neural Networks (DNNs)<n>It significantly decreases the time required for pruning by reducing the overall cost of fine-tuning.<n>ICE-Pruning can accelerate pruning by up to 9.61x.
• Score: 5.107302670511175
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Pruning is a widely used method for compressing Deep Neural Networks (DNNs), where less relevant parameters are removed from a DNN model to reduce its size. However, removing parameters reduces model accuracy, so pruning is typically combined with fine-tuning, and sometimes other operations such as rewinding weights, to recover accuracy. A common approach is to repeatedly prune and then fine-tune, with increasing amounts of model parameters being removed in each step. While straightforward to implement, pruning pipelines that follow this approach are computationally expensive due to the need for repeated fine-tuning. In this paper we propose ICE-Pruning, an iterative pruning pipeline for DNNs that significantly decreases the time required for pruning by reducing the overall cost of fine-tuning, while maintaining a similar accuracy to existing pruning pipelines. ICE-Pruning is based on three main components: i) an automatic mechanism to determine after which pruning steps fine-tuning should be performed; ii) a freezing strategy for faster fine-tuning in each pruning step; and iii) a custom pruning-aware learning rate scheduler to further improve the accuracy of each pruning step and reduce the overall time consumption. We also propose an efficient auto-tuning stage for the hyperparameters (e.g., freezing percentage) introduced by the three components. We evaluate ICE-Pruning on several DNN models and datasets, showing that it can accelerate pruning by up to 9.61x. Code is available at https://github.com/gicLAB/ICE-Pruning

Related papers

• Loss-Aware Automatic Selection of Structured Pruning Criteria for Deep Neural Network Acceleration [1.3225694028747144]
  This paper presents an efficient Loss-Aware Automatic Selection of Structured Pruning Criteria (LAASP) for slimming and accelerating deep neural networks.<n>The pruning-while-training approach eliminates the first stage and integrates the second and third stages into a single cycle.<n>Experiments on the VGGNet and ResNet models on the CIFAR-10 and ImageNet benchmark datasets demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2025-06-25T06:18:46Z)
• DRIVE: Dual Gradient-Based Rapid Iterative Pruning [2.209921757303168]
  Modern deep neural networks (DNNs) consist of millions of parameters, necessitating high-performance computing during training and inference. Traditional pruning methods that are applied post-training focus on streamlining inference, but there are recent efforts to leverage sparsity early on by pruning before training. We present Dual Gradient-Based Rapid Iterative Pruning (DRIVE), which leverages dense training for initial epochs to counteract the randomness inherent at the inception.
  arXiv  Detail & Related papers  (2024-04-01T20:44:28Z)
• Winner-Take-All Column Row Sampling for Memory Efficient Adaptation of Language Model [89.8764435351222]
  We propose a new family of unbiased estimators called WTA-CRS, for matrix production with reduced variance. Our work provides both theoretical and experimental evidence that, in the context of tuning transformers, our proposed estimators exhibit lower variance compared to existing ones.
  arXiv  Detail & Related papers  (2023-05-24T15:52:08Z)
• Boosting Pruned Networks with Linear Over-parameterization [8.796518772724955]
  Structured pruning compresses neural networks by reducing channels (filters) for fast inference and low footprint at run-time. To restore accuracy after pruning, fine-tuning is usually applied to pruned networks. We propose a novel method that first linearly over- parameterizes the compact layers in pruned networks to enlarge the number of fine-tuning parameters.
  arXiv  Detail & Related papers  (2022-04-25T05:30:26Z)
• Deep Equilibrium Optical Flow Estimation [80.80992684796566]
  Recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms. These RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. We propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer.
  arXiv  Detail & Related papers  (2022-04-18T17:53:44Z)
• A Fast Post-Training Pruning Framework for Transformers [74.59556951906468]
  Pruning is an effective way to reduce the huge inference cost of large Transformer models. Prior work on model pruning requires retraining the model. We propose a fast post-training pruning framework for Transformers that does not require any retraining.
  arXiv  Detail & Related papers  (2022-03-29T07:41:11Z)
• Interspace Pruning: Using Adaptive Filter Representations to Improve Training of Sparse CNNs [69.3939291118954]
  Unstructured pruning is well suited to reduce the memory footprint of convolutional neural networks (CNNs) Standard unstructured pruning (SP) reduces the memory footprint of CNNs by setting filter elements to zero. We introduce interspace pruning (IP), a general tool to improve existing pruning methods.
  arXiv  Detail & Related papers  (2022-03-15T11:50:45Z)
• Combined Pruning for Nested Cross-Validation to Accelerate Automated Hyperparameter Optimization for Embedded Feature Selection in High-Dimensional Data with Very Small Sample Sizes [3.51500332842165]
  Tree-based embedded feature selection to exclude irrelevant features in high-dimensional data with very small sample sizes requires optimized hyperparameters for the model building process. Standard pruning algorithms must prune late or risk aborting calculations due to high variance in the performance evaluation metric. We adapt the usage of a state-of-the-art successive halving pruner and combine it with two new pruning strategies based on domain or prior knowledge. Our proposed combined three-layer pruner keeps promising trials while reducing the number of models to be built by up to 81,3% compared to using a state-of-the-
  arXiv  Detail & Related papers  (2022-02-01T17:42:37Z)
• Pruning with Compensation: Efficient Channel Pruning for Deep Convolutional Neural Networks [0.9712140341805068]
  A highly efficient pruning method is proposed to significantly reduce the cost of pruning DCNN. Our method shows competitive pruning performance among the state-of-the-art retraining-based pruning methods.
  arXiv  Detail & Related papers  (2021-08-31T10:17:36Z)
• One-Cycle Pruning: Pruning ConvNets Under a Tight Training Budget [0.0]
  Introducing sparsity in a neural network has been an efficient way to reduce its complexity while keeping its performance almost intact. Most of the time, sparsity is introduced using a three-stage pipeline: 1) train the model to convergence, 2) prune the model according to some criterion, 3) fine-tune the pruned model to recover performance. In our work, we propose to get rid of the first step of the pipeline and to combine the two other steps in a single pruning-training cycle.
  arXiv  Detail & Related papers  (2021-07-05T15:27:07Z)
• MLPruning: A Multilevel Structured Pruning Framework for Transformer-based Models [78.45898846056303]
  Pruning is an effective method to reduce the memory footprint and computational cost associated with large natural language processing models. We develop a novel MultiLevel structured Pruning framework, which uses three different levels of structured pruning: head pruning, row pruning, and block-wise sparse pruning.
  arXiv  Detail & Related papers  (2021-05-30T22:00:44Z)
• Non-Parametric Adaptive Network Pruning [125.4414216272874]
  We introduce non-parametric modeling to simplify the algorithm design. Inspired by the face recognition community, we use a message passing algorithm to obtain an adaptive number of exemplars. EPruner breaks the dependency on the training data in determining the "important" filters.
  arXiv  Detail & Related papers  (2021-01-20T06:18:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.