Advancing Weight and Channel Sparsification with Enhanced Saliency

• URL: http://arxiv.org/abs/2502.03658v1
• Date: Wed, 05 Feb 2025 22:56:55 GMT
• Title: Advancing Weight and Channel Sparsification with Enhanced Saliency
• Authors: Xinglong Sun, Maying Shen, Hongxu Yin, Lei Mao, Pavlo Molchanov, Jose M. Alvarez,
• Abstract summary: Pruning aims to accelerate and compress models by removing redundant parameters. This removal is irreversible, often leading to subpar performance in pruned models. We introduce an efficient, innovative paradigm to enhance a given importance criterion for either unstructured or structured sparsity.
• Score: 27.89287351110155
• License:
• Abstract: Pruning aims to accelerate and compress models by removing redundant parameters, identified by specifically designed importance scores which are usually imperfect. This removal is irreversible, often leading to subpar performance in pruned models. Dynamic sparse training, while attempting to adjust sparse structures during training for continual reassessment and refinement, has several limitations including criterion inconsistency between pruning and growth, unsuitability for structured sparsity, and short-sighted growth strategies. Our paper introduces an efficient, innovative paradigm to enhance a given importance criterion for either unstructured or structured sparsity. Our method separates the model into an active structure for exploitation and an exploration space for potential updates. During exploitation, we optimize the active structure, whereas in exploration, we reevaluate and reintegrate parameters from the exploration space through a pruning and growing step consistently guided by the same given importance criterion. To prepare for exploration, we briefly "reactivate" all parameters in the exploration space and train them for a few iterations while keeping the active part frozen, offering a preview of the potential performance gains from reintegrating these parameters. We show on various datasets and configurations that existing importance criterion even simple as magnitude can be enhanced with ours to achieve state-of-the-art performance and training cost reductions. Notably, on ImageNet with ResNet50, ours achieves an +1.3 increase in Top-1 accuracy over prior art at 90% ERK sparsity. Compared with the SOTA latency pruning method HALP, we reduced its training cost by over 70% while attaining a faster and more accurate pruned model.

Related papers

• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts. Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• Pushing the Limits of Sparsity: A Bag of Tricks for Extreme Pruning [4.421875265386832]
  Pruning of deep neural networks has been an effective technique for reducing model size while preserving most of the performance of dense networks. Recent sparse learning methods have shown promising performance up to moderate sparsity levels such as 95% and 98%. We propose a collection of techniques that enable the continuous learning of networks without accuracy collapse even at extreme sparsities.
  arXiv  Detail & Related papers  (2024-11-20T18:54:53Z)
• SaRA: High-Efficient Diffusion Model Fine-tuning with Progressive Sparse Low-Rank Adaptation [52.6922833948127]
  In this work, we investigate the importance of parameters in pre-trained diffusion models. We propose a novel model fine-tuning method to make full use of these ineffective parameters. Our method enhances the generative capabilities of pre-trained models in downstream applications.
  arXiv  Detail & Related papers  (2024-09-10T16:44:47Z)
• Enhanced Sparsification via Stimulative Training [36.0559905521154]
  Existing methods commonly set sparsity-inducing penalty terms to suppress the importance of dropped weights. We propose a structured pruning framework, named expressivity, based on an enhanced sparsification paradigm. To reduce the huge capacity gap of distillation, we propose a mutating expansion technique.
  arXiv  Detail & Related papers  (2024-03-11T04:05:17Z)
• PERP: Rethinking the Prune-Retrain Paradigm in the Era of LLMs [22.557682089926004]
  We show that updating a small subset of parameters can suffice to recover or even enhance performance after pruning. We introduce two novel LoRA variants that, unlike standard LoRA, allow merging adapters back without compromising sparsity.
  arXiv  Detail & Related papers  (2023-12-23T11:45:22Z)
• FOSTER: Feature Boosting and Compression for Class-Incremental Learning [52.603520403933985]
  Deep neural networks suffer from catastrophic forgetting when learning new categories. We propose a novel two-stage learning paradigm FOSTER, empowering the model to learn new categories adaptively.
  arXiv  Detail & Related papers  (2022-04-10T11:38:33Z)
• Online Convolutional Re-parameterization [51.97831675242173]
  We present online convolutional re- parameterization (OREPA), a two-stage pipeline, aiming to reduce the huge training overhead by squeezing the complex training-time block into a single convolution. Compared with the state-of-the-art re-param models, OREPA is able to save the training-time memory cost by about 70% and accelerate the training speed by around 2x. We also conduct experiments on object detection and semantic segmentation and show consistent improvements on the downstream tasks.
  arXiv  Detail & Related papers  (2022-04-02T09:50:19Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)
• Progressive Skeletonization: Trimming more fat from a network at initialization [76.11947969140608]
  We propose an objective to find a skeletonized network with maximum connection sensitivity. We then propose two approximate procedures to maximize our objective. Our approach provides remarkably improved performance on higher pruning levels.
  arXiv  Detail & Related papers  (2020-06-16T11:32:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.