Towards Higher Ranks via Adversarial Weight Pruning
- URL: http://arxiv.org/abs/2311.17493v1
- Date: Wed, 29 Nov 2023 10:04:39 GMT
- Title: Towards Higher Ranks via Adversarial Weight Pruning
- Authors: Yuchuan Tian, Hanting Chen, Tianyu Guo, Chao Xu, Yunhe Wang
- Abstract summary: We propose a Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an adversarial manner.
RPG outperforms the state-of-the-art performance by 1.13% top-1 accuracy on ImageNet in ResNet-50 with 98% sparsity.
- Score: 34.602137305496335
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Convolutional Neural Networks (CNNs) are hard to deploy on edge devices due
to its high computation and storage complexities. As a common practice for
model compression, network pruning consists of two major categories:
unstructured and structured pruning, where unstructured pruning constantly
performs better. However, unstructured pruning presents a structured pattern at
high pruning rates, which limits its performance. To this end, we propose a
Rank-based PruninG (RPG) method to maintain the ranks of sparse weights in an
adversarial manner. In each step, we minimize the low-rank approximation error
for the weight matrices using singular value decomposition, and maximize their
distance by pushing the weight matrices away from its low rank approximation.
This rank-based optimization objective guides sparse weights towards a
high-rank topology. The proposed method is conducted in a gradual pruning
fashion to stabilize the change of rank during training. Experimental results
on various datasets and different tasks demonstrate the effectiveness of our
algorithm in high sparsity. The proposed RPG outperforms the state-of-the-art
performance by 1.13% top-1 accuracy on ImageNet in ResNet-50 with 98% sparsity.
The codes are available at
https://github.com/huawei-noah/Efficient-Computing/tree/master/Pruning/RPG and
https://gitee.com/mindspore/models/tree/master/research/cv/RPG.
Related papers
- InRank: Incremental Low-Rank Learning [85.6380047359139]
gradient-based training implicitly regularizes neural networks towards low-rank solutions through a gradual increase of the rank during training.
Existing training algorithms do not exploit the low-rank property to improve computational efficiency.
We design a new training algorithm Incremental Low-Rank Learning (InRank), which explicitly expresses cumulative weight updates as low-rank matrices.
arXiv Detail & Related papers (2023-06-20T03:03:04Z) - What to Prune and What Not to Prune at Initialization [0.0]
Post-training dropout based approaches achieve high sparsity.
Initialization pruning is more efficacious when it comes to scaling computation cost of the network.
The goal is to achieve higher sparsity while preserving performance.
arXiv Detail & Related papers (2022-09-06T03:48:10Z) - FairGRAPE: Fairness-aware GRAdient Pruning mEthod for Face Attribute
Classification [4.909402570564468]
We propose a novel pruning method, Fairness-aware GRAdient Pruning mEthod (FairGRAPE)
Our method calculates the per-group importance of each model weight and selects a subset of weights that maintain the relative between-group total importance in pruning.
Our method is substantially more effective in a setting with a high pruning rate (99%)
arXiv Detail & Related papers (2022-07-22T05:44:03Z) - Monarch: Expressive Structured Matrices for Efficient and Accurate
Training [64.6871423399431]
Large neural networks excel in many domains, but they are expensive to train and fine-tune.
A popular approach to reduce their compute or memory requirements is to replace dense weight matrices with structured ones.
We propose a class of matrices (Monarch) that is hardware-efficient.
arXiv Detail & Related papers (2022-04-01T17:37:29Z) - GNNRank: Learning Global Rankings from Pairwise Comparisons via Directed
Graph Neural Networks [68.61934077627085]
We introduce GNNRank, a modeling framework compatible with any GNN capable of learning digraph embeddings.
We show that our methods attain competitive and often superior performance compared with existing approaches.
arXiv Detail & Related papers (2022-02-01T04:19:50Z) - Scaling Structured Inference with Randomization [64.18063627155128]
We propose a family of dynamic programming (RDP) randomized for scaling structured models to tens of thousands of latent states.
Our method is widely applicable to classical DP-based inference.
It is also compatible with automatic differentiation so can be integrated with neural networks seamlessly.
arXiv Detail & Related papers (2021-12-07T11:26:41Z) - Structured Directional Pruning via Perturbation Orthogonal Projection [13.704348351073147]
A more reasonable approach is to find a sparse minimizer along the flat minimum valley found byNIST.
We propose the structured directional pruning based on projecting the perturbations onto the flat minimum valley.
Experiments show that our method obtains the state-of-the-art pruned accuracy (i.e. 93.97% on VGG16, CIFAR-10 task) without retraining.
arXiv Detail & Related papers (2021-07-12T11:35:47Z) - Effective Model Sparsification by Scheduled Grow-and-Prune Methods [73.03533268740605]
We propose a novel scheduled grow-and-prune (GaP) methodology without pre-training the dense models.
Experiments have shown that such models can match or beat the quality of highly optimized dense models at 80% sparsity on a variety of tasks.
arXiv Detail & Related papers (2021-06-18T01:03:13Z) - Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch [75.69506249886622]
Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments.
In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network.
arXiv Detail & Related papers (2021-02-08T05:55:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.