One-cycle Structured Pruning with Stability Driven Structure Search

• URL: http://arxiv.org/abs/2501.13439v1
• Date: Thu, 23 Jan 2025 07:46:48 GMT
• Title: One-cycle Structured Pruning with Stability Driven Structure Search
• Authors: Deepak Ghimire, Dayoung Kil, Seonghwan Jeong, Jaesik Park, Seong-heum Kim,
• Abstract summary: Existing structured pruning typically involves multi-stage training procedures that often demand heavy computation. We propose an efficient framework for one-cycle structured pruning without compromising model performance. Our method achieves state-of-the-art accuracy while being one of the most efficient pruning frameworks in terms of training time.
• Score: 20.18712941647407
• License:
• Abstract: Existing structured pruning typically involves multi-stage training procedures that often demand heavy computation. Pruning at initialization, which aims to address this limitation, reduces training costs but struggles with performance. To address these challenges, we propose an efficient framework for one-cycle structured pruning without compromising model performance. In this approach, we integrate pre-training, pruning, and fine-tuning into a single training cycle, referred to as the `one cycle approach'. The core idea is to search for the optimal sub-network during the early stages of network training, guided by norm-based group saliency criteria and structured sparsity regularization. We introduce a novel pruning indicator that determines the stable pruning epoch by assessing the similarity between evolving pruning sub-networks across consecutive training epochs. Also, group sparsity regularization helps to accelerate the pruning process and results in speeding up the entire process. Extensive experiments on datasets, including CIFAR-10/100, and ImageNet, using VGGNet, ResNet, MobileNet, and ViT architectures, demonstrate that our method achieves state-of-the-art accuracy while being one of the most efficient pruning frameworks in terms of training time. The source code will be made publicly available.

Related papers

• Structurally Prune Anything: Any Architecture, Any Framework, Any Time [84.6210631783801]
  We introduce Structurally Prune Anything (SPA), a versatile structured pruning framework for neural networks. SPA supports pruning at any time, either before training, after training with fine-tuning, or after training without fine-tuning. In extensive experiments, SPA shows competitive to state-of-the-art pruning performance across various architectures.
  arXiv  Detail & Related papers  (2024-03-03T13:49:49Z)
• Efficient Stagewise Pretraining via Progressive Subnetworks [53.00045381931778]
  The prevailing view suggests that stagewise dropping strategies, such as layer dropping, are ineffective when compared to stacking-based approaches. This paper challenges this notion by demonstrating that, with proper design, dropping strategies can be competitive, if not better, than stacking methods. We propose an instantiation of this framework - Random Part Training (RAPTR) - that selects and trains only a random subnetwork at each step, progressively increasing the size in stages.
  arXiv  Detail & Related papers  (2024-02-08T18:49:09Z)
• ThinResNet: A New Baseline for Structured Convolutional Networks Pruning [1.90298817989995]
  Pruning is a compression method which aims to improve the efficiency of neural networks by reducing their number of parameters. In this work, we verify how results in the recent literature of pruning hold up against networks that underwent both state-of-the-art training methods and trivial model scaling.
  arXiv  Detail & Related papers  (2023-09-22T13:28:18Z)
• Fantastic Weights and How to Find Them: Where to Prune in Dynamic Sparse Training [58.47622737624532]
  We study the influence of pruning criteria on Dynamic Sparse Training (DST) performance. We find that most of the studied methods yield similar results. The best performance is predominantly given by the simplest technique: magnitude-based pruning.
  arXiv  Detail & Related papers  (2023-06-21T12:43:55Z)
• A Unified Framework for Soft Threshold Pruning [27.853698217792456]
  We reformulate soft threshold pruning as an implicit optimization problem solved using the Iterative Shrinkage-Thresholding Algorithm (ISTA) We derive an optimal threshold scheduler through an in-depth study of threshold scheduling based on our framework. In principle, the derived pruning algorithm could sparsify any mathematical model trained via SGD.
  arXiv  Detail & Related papers  (2023-02-25T08:16:14Z)
• Pushing the Efficiency Limit Using Structured Sparse Convolutions [82.31130122200578]
  We propose Structured Sparse Convolution (SSC), which leverages the inherent structure in images to reduce the parameters in the convolutional filter. We show that SSC is a generalization of commonly used layers (depthwise, groupwise and pointwise convolution) in efficient architectures'' Architectures based on SSC achieve state-of-the-art performance compared to baselines on CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet classification benchmarks.
  arXiv  Detail & Related papers  (2022-10-23T18:37:22Z)
• Back to Basics: Efficient Network Compression via IMP [22.586474627159287]
  Iterative Magnitude Pruning (IMP) is one of the most established approaches for network pruning. IMP is often argued that it reaches suboptimal states by not incorporating sparsification into the training phase. We find that IMP with SLR for retraining can outperform state-of-the-art pruning-during-training approaches.
  arXiv  Detail & Related papers  (2021-11-01T11:23:44Z)
• When to Prune? A Policy towards Early Structural Pruning [27.91996628143805]
  We propose a policy that prunes as early as possible during training without hurting performance. Our method yields $1.4%$ top-1 accuracy boost over state-of-the-art pruning counterparts, cuts down training cost on GPU by $2.4times$.
  arXiv  Detail & Related papers  (2021-10-22T18:39:22Z)
• Rapid Structural Pruning of Neural Networks with Set-based Task-Adaptive Meta-Pruning [83.59005356327103]
  A common limitation of most existing pruning techniques is that they require pre-training of the network at least once before pruning. We propose STAMP, which task-adaptively prunes a network pretrained on a large reference dataset by generating a pruning mask on it as a function of the target dataset. We validate STAMP against recent advanced pruning methods on benchmark datasets.
  arXiv  Detail & Related papers  (2020-06-22T10:57:43Z)
• Subset Sampling For Progressive Neural Network Learning [106.12874293597754]
  Progressive Neural Network Learning is a class of algorithms that incrementally construct the network's topology and optimize its parameters based on the training data. We propose to speed up this process by exploiting subsets of training data at each incremental training step. Experimental results in object, scene and face recognition problems demonstrate that the proposed approach speeds up the optimization procedure considerably.
  arXiv  Detail & Related papers  (2020-02-17T18:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.