$\rm SP^3$: Enhancing Structured Pruning via PCA Projection

• URL: http://arxiv.org/abs/2308.16475v3
• Date: Sun, 18 Aug 2024 04:08:29 GMT
• Title: $\rm SP^3$: Enhancing Structured Pruning via PCA Projection
• Abstract summary: This paper introduces a novel structured pruning approach, Structured Pruning with PCA Projection (SP3), targeting the effective reduction of d. Experiments show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, maintain over 96% accuracy, and outperform other methods that compress d by 6% in accuracy at the same compression ratio.
• Score: 24.74632458058136
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Structured pruning is a widely used technique for reducing the size of pre-trained language models (PLMs), but current methods often overlook the potential of compressing the hidden dimension (d) in PLMs, a dimension critical to model size and efficiency. This paper introduces a novel structured pruning approach, Structured Pruning with PCA Projection (SP3), targeting the effective reduction of d by projecting features into a space defined by principal components before masking. Extensive experiments on benchmarks (GLUE and SQuAD) show that SP3 can reduce d by 70%, compress 94% of the BERTbase model, maintain over 96% accuracy, and outperform other methods that compress d by 6% in accuracy at the same compression ratio. SP3 has also proven effective with other models, including OPT and Llama. Our data and code are available at an anonymous repo.

Related papers

• PSViT: A Methodology for Structurally Pruning Spiking Vision Transformers [2.9583391652768314]
  Spiking Vision Transformer (SViT) models are promising low-power ViT models for solving vision-based tasks.<n>We propose PSViT, a novel methodology to perform structured pruning on SViT models.
  arXiv  Detail & Related papers  (2026-06-02T07:18:57Z)
• OFA-Diffusion Compression: Compressing Diffusion Model in One-Shot Manner [19.771011835343653]
  Diffusion Probabilistic Model (DPM) achieves remarkable performance in image generation.<n>Existing literature focuses on obtaining a smaller model with a fixed architecture through model compression.<n>We propose a once-for-all (OFA) compression framework for DPMs that yields differentworks with various computations in a one-shot training manner.
  arXiv  Detail & Related papers  (2026-04-14T12:39:13Z)
• Lossless Compression of Neural Network Components: Weights, Checkpoints, and K/V Caches in Low-Precision Formats [0.0]
  In this work, we extend the ZipNN approach to lower-precision floating-point formats, specifically FP8 and FP4.<n>Our evaluation shows compression ratios up to 62% for BF16 and 83% for FP8.<n>We also investigate the compressibility of key-value (K/V) cache tensors used in large language models.
  arXiv  Detail & Related papers  (2025-08-20T12:46:50Z)
• You Only Prune Once: Designing Calibration-Free Model Compression With Policy Learning [20.62274005080048]
  PruneNet is a novel model compression method that reformulates model pruning as a policy learning process. It can compress the LLaMA-2-7B model in just 15 minutes, achieving over 80% retention of its zero-shot performance. On complex multitask language understanding tasks, PruneNet demonstrates its robustness by preserving up to 80% performance of the original model.
  arXiv  Detail & Related papers  (2025-01-25T18:26:39Z)
• PIP: Perturbation-based Iterative Pruning for Large Language Models [5.511065308044068]
  We propose PIP (Perturbation-based Iterative Pruning), a novel double-view structured pruning method to optimize Large Language Models. Our experiments show that PIP reduces the parameter count by approximately 20% while retaining over 85% of the original model's accuracy.
  arXiv  Detail & Related papers  (2025-01-25T17:10:50Z)
• FlexiGPT: Pruning and Extending Large Language Models with Low-Rank Weight Sharing [59.12511498024836]
  We present a method to prune large language models (LLMs) that selectively prunes model blocks based on an importance score. We propose a principled metric to replace each pruned block using a weight-sharing mechanism. Empirical evaluations demonstrate substantial performance gains over existing methods.
  arXiv  Detail & Related papers  (2025-01-24T18:46:37Z)
• ALoRE: Efficient Visual Adaptation via Aggregating Low Rank Experts [71.91042186338163]
  ALoRE is a novel PETL method that reuses the hypercomplex parameterized space constructed by Kronecker product to Aggregate Low Rank Experts. Thanks to the artful design, ALoRE maintains negligible extra parameters and can be effortlessly merged into the frozen backbone.
  arXiv  Detail & Related papers  (2024-12-11T12:31:30Z)
• MatryoshkaKV: Adaptive KV Compression via Trainable Orthogonal Projection [14.073722038551125]
  KV cache has become a de facto technique for the inference of large language models. This paper uses low-rank projection matrices to transform the cache features into spaces with reduced dimensions. We find that our method can sustain over 90% performance with an average KV cache compression rate of 60%.
  arXiv  Detail & Related papers  (2024-10-16T08:34:51Z)
• Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism [25.36736897890854]
  Pre-trained language models (PLMs) are engineered to be robust in contextual understanding and exhibit outstanding performance in various natural language processing tasks. Modern pruning strategies employ one-shot techniques to compress PLMs without the need for retraining on task-specific or otherwise general data. We propose SDS, a Sparse-Dense-Sparse pruning framework to enhance the performance of the pruned PLMs from a weight distribution optimization perspective.
  arXiv  Detail & Related papers  (2024-08-20T01:05:45Z)
• MoDeGPT: Modular Decomposition for Large Language Model Compression [59.361006801465344]
  This paper introduces textbfModular bfDecomposition (MoDeGPT), a novel structured compression framework. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions. Our experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods.
  arXiv  Detail & Related papers  (2024-08-19T01:30:14Z)
• Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging [14.123313596780726]
  We propose Manifold-Based Knowledge Alignment and Layer Merging Compression (MKA) MKA uses manifold learning and the Normalized Pairwise Information Bottleneck measure to merge similar layers, reducing model size while preserving essential performance. Our findings show that MKA not only preserves model performance but also achieves substantial compression ratios, outperforming traditional pruning methods.
  arXiv  Detail & Related papers  (2024-06-24T05:57:55Z)
• Bypass Back-propagation: Optimization-based Structural Pruning for Large Language Models via Policy Gradient [57.9629676017527]
  We propose an optimization-based structural pruning on Large-Language Models. We learn the pruning masks in a probabilistic space directly by optimizing the loss of the pruned model. Our method operates for 2.7 hours with around 35GB memory for the 13B models on a single A100 GPU.
  arXiv  Detail & Related papers  (2024-06-15T09:31:03Z)
• SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Models [53.638791265113625]
  Sparsity-Preserved efficient fine-tuning method for large language models. Code will be made available at https://github.com/Lucky-Lance/SPP.
  arXiv  Detail & Related papers  (2024-05-25T04:55:27Z)
• A Simple Baseline for Efficient Hand Mesh Reconstruction [9.704302427105187]
  3D hand pose estimation has found broad application in areas such as gesture recognition and human-machine interaction tasks. We propose a simple yet effective baseline that surpasses state-of-the-art (SOTA) methods but also demonstrates computational efficiency.
  arXiv  Detail & Related papers  (2024-03-04T08:00:20Z)
• Data-freeWeight Compress and Denoise for Large Language Models [101.53420111286952]
  We propose a novel approach termed Data-free Joint Rank-k Approximation for compressing the parameter matrices. We achieve a model pruning of 80% parameters while retaining 93.43% of the original performance without any calibration data.
  arXiv  Detail & Related papers  (2024-02-26T05:51:47Z)
• Language model compression with weighted low-rank factorization [73.61874728240568]
  We introduce Fisher information to weigh the importance of parameters affecting the model prediction. We find that our resulting task accuracy is much closer to the original model's performance. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies.
  arXiv  Detail & Related papers  (2022-06-30T21:57:07Z)
• Automatic Mixed-Precision Quantization Search of BERT [62.65905462141319]
  Pre-trained language models such as BERT have shown remarkable effectiveness in various natural language processing tasks. These models usually contain millions of parameters, which prevents them from practical deployment on resource-constrained devices. We propose an automatic mixed-precision quantization framework designed for BERT that can simultaneously conduct quantization and pruning in a subgroup-wise level.
  arXiv  Detail & Related papers  (2021-12-30T06:32:47Z)
• Sparse Tensor-based Multiscale Representation for Point Cloud Geometry Compression [18.24902526033056]
  We develop a unified Point Cloud Geometry (PCG) compression method through Sparse Processing (STP) based multiscale representation of voxelized PCG. Applying the complexity reduces the complexity significantly because it only performs the convolutions centered at Most-Probable Positively-Occupied Voxels (MP-POV) The proposed method presents lightweight complexity due to point-wise, and tiny storage desire because of model sharing across all scales.
  arXiv  Detail & Related papers  (2021-11-20T17:02:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.