Delta Decompression for MoE-based LLMs Compression

• URL: http://arxiv.org/abs/2502.17298v1
• Date: Mon, 24 Feb 2025 16:32:22 GMT
• Title: Delta Decompression for MoE-based LLMs Compression
• Authors: Hao Gu, Wei Li, Lujun Li, Qiyuan Zhu, Mark Lee, Shengjie Sun, Wei Xue, Yike Guo,
• Abstract summary: $D2$-MoE is a new delta decompression compressor for reducing the parameters of MoE LLMs.<n>We decompose their weights into a shared base weight and unique delta weights.<n>Experiments highlight the superiority of our approach, with over 13% performance gains.
• Score: 22.144081182788394
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Mixture-of-Experts (MoE) architectures in large language models (LLMs) achieve exceptional performance, but face prohibitive storage and memory requirements. To address these challenges, we present $D^2$-MoE, a new delta decompression compressor for reducing the parameters of MoE LLMs. Based on observations of expert diversity, we decompose their weights into a shared base weight and unique delta weights. Specifically, our method first merges each expert's weight into the base weight using the Fisher information matrix to capture shared components. Then, we compress delta weights through Singular Value Decomposition (SVD) by exploiting their low-rank properties. Finally, we introduce a semi-dynamical structured pruning strategy for the base weights, combining static and dynamic redundancy analysis to achieve further parameter reduction while maintaining input adaptivity. In this way, our $D^2$-MoE successfully compact MoE LLMs to high compression ratios without additional training. Extensive experiments highlight the superiority of our approach, with over 13% performance gains than other compressors on Mixtral|Phi-3.5|DeepSeek|Qwen2 MoE LLMs at 40$\sim$60% compression rates. Codes are available in https://github.com/lliai/D2MoE.

Related papers

• ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restoration [61.579842548990754]
  Mixture-of-Experts (MoE) Transformer, the backbone of multiple phenomenal language models, leverages sparsity by activating only a fraction of model parameters for each input token. We introduce ResMoE, an innovative MoE approximation framework that utilizes Wasserstein barycenter to extract a common expert (barycenter expert) and approximate the residuals between this barycenter expert and the original ones.
  arXiv  Detail & Related papers  (2025-03-10T03:15:54Z)
• DeltaLLM: Compress LLMs with Low-Rank Deltas between Shared Weights [11.047879241587315]
  We introduce DeltaLLM, a new post-training compression technique to reduce the memory footprint of LLMs.<n>For training, we adopt the progressing module replacement method and show that the lightweight training of the low-rank modules is sufficient to achieve performance on par with LLMs of comparable sizes trained from scratch.<n>Our method also outperforms compression techniques JointDrop, LaCo, ShortGPT and SliceGPT with the same number of parameters removed.
  arXiv  Detail & Related papers  (2025-01-30T18:59:55Z)
• Lillama: Large Language Models Compression via Low-Rank Feature Distillation [8.090496457850852]
  Lillama is a compression method that distills activations with low-rank weights.<n>It compresses Mixtral-8x7B within minutes on a single A100 GPU, removing 10 billion parameters while retaining over 95% of its original performance.<n>It generalizes well to non-transformer architectures, compressing Mamba-3B by 20% while maintaining 99% performance.
  arXiv  Detail & Related papers  (2024-12-21T18:04:01Z)
• EoRA: Training-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation [84.70637613266835]
  We re-formulate the model compression problem into the customized compensation problem. We propose Training-free Eigenspace Low-Rank Approximation (EoRA) EoRA directly minimizes compression-induced errors without requiring gradient-based training.
  arXiv  Detail & Related papers  (2024-10-28T17:59:03Z)
• DeltaDQ: Ultra-High Delta Compression for Fine-Tuned LLMs via Group-wise Dropout and Separate Quantization [17.501956455837707]
  Large language models achieve exceptional performance on various downstream tasks through supervised fine-tuning. Current methods that compress the delta weight struggle to achieve ultra-high compression. We propose a novel distribution-driven delta compression framework DeltaDQ to achieve ultra-high compression for the delta weight.
  arXiv  Detail & Related papers  (2024-10-11T09:44:16Z)
• Basis Sharing: Cross-Layer Parameter Sharing for Large Language Model Compression [5.206085750261924]
  Large Language Models (LLMs) require significant amount of memory storage in inference. In this paper, we take a step further to explore parameter sharing across different layers with singular value decomposition. Comprehensive experiments demonstrate that Basis Sharing outperforms state-of-the-art SVD-based compression approaches.
  arXiv  Detail & Related papers  (2024-10-02T14:30:02Z)
• MoDeGPT: Modular Decomposition for Large Language Model Compression [59.361006801465344]
  This paper introduces textbfModular bfDecomposition (MoDeGPT), a novel structured compression framework. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions. Our experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods.
  arXiv  Detail & Related papers  (2024-08-19T01:30:14Z)
• Pruning via Merging: Compressing LLMs via Manifold Alignment Based Layer Merging [14.123313596780726]
  We propose Manifold-Based Knowledge Alignment and Layer Merging Compression (MKA) MKA uses manifold learning and the Normalized Pairwise Information Bottleneck measure to merge similar layers, reducing model size while preserving essential performance. Our findings show that MKA not only preserves model performance but also achieves substantial compression ratios, outperforming traditional pruning methods.
  arXiv  Detail & Related papers  (2024-06-24T05:57:55Z)
• SVD-LLM: Truncation-aware Singular Value Decomposition for Large Language Model Compression [14.818355326032538]
  Singular Value Decomposition (SVD) offers a promising solution for Large Language Models (LLMs) compression. However, truncating smaller singular values may lead to higher compression loss, and the lack of update on the compressed weights after SVD truncation. We propose SVD-LLM, a SVD-based post-training LLM compression method that addresses the limitations of existing methods.
  arXiv  Detail & Related papers  (2024-03-12T07:31:18Z)
• Head-wise Shareable Attention for Large Language Models [56.92068213969036]
  Large Language Models (LLMs) suffer from huge number of parameters, which restricts their deployment on edge devices. Weight sharing is one promising solution that encourages weight reuse, effectively reducing memory usage with less performance drop. We present a perspective on head-wise shareable attention for large language models.
  arXiv  Detail & Related papers  (2024-02-19T04:19:36Z)
• Compressing LLMs: The Truth is Rarely Pure and Never Simple [90.05366363633568]
  Knowledge-Intensive Compressed LLM BenchmarK aims to redefine the evaluation protocol for compressed Large Language Models. LLM-KICK unveils many favorable merits and unfortunate plights of current SoTA compression methods. LLM-KICK is designed to holistically access compressed LLMs' ability for language understanding, reasoning, generation, in-context retrieval, in-context summarization, etc.
  arXiv  Detail & Related papers  (2023-10-02T17:42:37Z)
• Towards Compact CNNs via Collaborative Compression [166.86915086497433]
  We propose a Collaborative Compression scheme, which joints channel pruning and tensor decomposition to compress CNN models. We achieve 52.9% FLOPs reduction by removing 48.4% parameters on ResNet-50 with only a Top-1 accuracy drop of 0.56% on ImageNet 2012.
  arXiv  Detail & Related papers  (2021-05-24T12:07:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.