Low-Rank Matrix Approximation for Neural Network Compression

• URL: http://arxiv.org/abs/2504.20078v1
• Date: Fri, 25 Apr 2025 06:04:01 GMT
• Title: Low-Rank Matrix Approximation for Neural Network Compression
• Authors: Kalyan Cherukuri, Aarav Lala,
• Abstract summary: We introduce a novel adaptive-rank Singular Value Decomposition (ARSVD)<n>ARSVD dynamically chooses the rank increase of the fully connected layers below a certain threshold in energy expenditure.<n>Our results demonstrate that the technique can achieve substantial model compression without compromising classification accuracy.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep Neural Networks (DNNs) are often constrained by their large memories and computational restrictions. In this paper, we introduce a novel adaptive-rank Singular Value Decomposition (ARSVD) that dynamically chooses the rank increase of the fully connected layers below a certain threshold in energy expenditure. Unlike conventional SVD compression methods that apply a fixed rank reduction in all layers, our ARSVD method uses energy distribution to adaptively select rank per layer while retaining accuracy. This is done for each layer in an effort to use as much energy as possible while maintaining the lowest accuracy loss. Such accuracy-adaptive approaches outperform traditional static rank reduction methods by providing an improved balance between compression and model performance. We first train a simple Multi-Layer Perceptron (MLP) on the MNIST, CIFAR-10, and CIFAR-100 dataset and evaluate its performance using accuracy and F1-score. After applying ARSVD, our results demonstrate that the technique can achieve substantial model compression without compromising classification accuracy. These results illustrate the usefulness of ARSVD in computing scenarios where both computational and memory resources are scarce.

Related papers

• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models [56.00251589760559]
  Large language models (LLMs) can act as gradient priors in a zero-shot setting.<n>We introduce LM-GC, a novel method that integrates LLMs with arithmetic coding.<n>Experiments indicate that LM-GC surpasses existing state-of-the-art lossless compression methods.
  arXiv  Detail & Related papers  (2024-09-26T13:38:33Z)
• Convolutional Neural Network Compression Based on Low-Rank Decomposition [3.3295360710329738]
  This paper proposes a model compression method that integrates Variational Bayesian Matrix Factorization. VBMF is employed to estimate the rank of the weight tensor at each layer. Experimental results show that for both high and low compression ratios, our compression model exhibits advanced performance.
  arXiv  Detail & Related papers  (2024-08-29T06:40:34Z)
• Structure-Preserving Network Compression Via Low-Rank Induced Training Through Linear Layers Composition [11.399520888150468]
  We present a theoretically-justified technique termed Low-Rank Induced Training (LoRITa) LoRITa promotes low-rankness through the composition of linear layers and compresses by using singular value truncation. We demonstrate the effectiveness of our approach using MNIST on Fully Connected Networks, CIFAR10 on Vision Transformers, and CIFAR10/100 and ImageNet on Convolutional Neural Networks.
  arXiv  Detail & Related papers  (2024-05-06T00:58:23Z)
• Improving Covariance Conditioning of the SVD Meta-layer by Orthogonality [65.67315418971688]
  Nearest Orthogonal Gradient (NOG) and Optimal Learning Rate (OLR) are proposed. Experiments on visual recognition demonstrate that our methods can simultaneously improve the covariance conditioning and generalization.
  arXiv  Detail & Related papers  (2022-07-05T15:39:29Z)
• BiTAT: Neural Network Binarization with Task-dependent Aggregated Transformation [116.26521375592759]
  Quantization aims to transform high-precision weights and activations of a given neural network into low-precision weights/activations for reduced memory usage and computation. Extreme quantization (1-bit weight/1-bit activations) of compactly-designed backbone architectures results in severe performance degeneration. This paper proposes a novel Quantization-Aware Training (QAT) method that can effectively alleviate performance degeneration.
  arXiv  Detail & Related papers  (2022-07-04T13:25:49Z)
• FOSTER: Feature Boosting and Compression for Class-Incremental Learning [52.603520403933985]
  Deep neural networks suffer from catastrophic forgetting when learning new categories. We propose a novel two-stage learning paradigm FOSTER, empowering the model to learn new categories adaptively.
  arXiv  Detail & Related papers  (2022-04-10T11:38:33Z)
• Learning Robust and Lightweight Model through Separable Structured Transformations [13.208781763887947]
  We propose a separable structural transformation of the fully-connected layer to reduce the parameters of convolutional neural networks. We successfully reduce the amount of network parameters by 90%, while the robust accuracy loss is less than 1.5%. We evaluate the proposed approach on datasets such as ImageNet, SVHN, CIFAR-100 and Vision Transformer.
  arXiv  Detail & Related papers  (2021-12-27T07:25:26Z)
• Low-rank Tensor Decomposition for Compression of Convolutional Neural Networks Using Funnel Regularization [1.8579693774597708]
  We propose a model reduction method to compress the pre-trained networks using low-rank tensor decomposition. A new regularization method, called funnel function, is proposed to suppress the unimportant factors during the compression. For ResNet18 with ImageNet2012, our reduced model can reach more than twi times speed up in terms of GMAC with merely 0.7% Top-1 accuracy drop.
  arXiv  Detail & Related papers  (2021-12-07T13:41:51Z)
• Sharpness-aware Quantization for Deep Neural Networks [45.150346855368]
  Sharpness-Aware Quantization (SAQ) is a novel method to explore the effect of Sharpness-Aware Minimization (SAM) on model compression. We show that SAQ improves the generalization performance of the quantized models, yielding the SOTA results in uniform quantization.
  arXiv  Detail & Related papers  (2021-11-24T05:16:41Z)
• An Efficient Statistical-based Gradient Compression Technique for Distributed Training Systems [77.88178159830905]
  Sparsity-Inducing Distribution-based Compression (SIDCo) is a threshold-based sparsification scheme that enjoys similar threshold estimation quality to deep gradient compression (DGC) Our evaluation shows SIDCo speeds up training by up to 41:7%, 7:6%, and 1:9% compared to the no-compression baseline, Topk, and DGC compressors, respectively.
  arXiv  Detail & Related papers  (2021-01-26T13:06:00Z)
• A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs [8.597091257152567]
  We present a dynamic network rewiring (DNR) method to generate pruned deep neural network (DNN) models that are robust against adversarial attacks. Our experiments show that DNR consistently finds compressed models with better clean and adversarial image classification performance than what is achievable through state-of-the-art alternatives.
  arXiv  Detail & Related papers  (2020-11-03T19:49:00Z)
• Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsification [53.50708351813565]
  We propose SVD training, the first method to explicitly achieve low-rank DNNs during training without applying SVD on every step. We empirically show that SVD training can significantly reduce the rank of DNN layers and achieve higher reduction on computation load under the same accuracy.
  arXiv  Detail & Related papers  (2020-04-20T02:40:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.