MoDeGPT: Modular Decomposition for Large Language Model Compression

• URL: http://arxiv.org/abs/2408.09632v3
• Date: Fri, 13 Sep 2024 05:34:14 GMT
• Title: MoDeGPT: Modular Decomposition for Large Language Model Compression
• Authors: Chi-Heng Lin, Shangqian Gao, James Seale Smith, Abhishek Patel, Shikhar Tuli, Yilin Shen, Hongxia Jin, Yen-Chang Hsu,
• Abstract summary: This paper introduces textbfModular bfDecomposition (MoDeGPT), a novel structured compression framework. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions. Our experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods.
• Score: 59.361006801465344
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) have reshaped the landscape of artificial intelligence by demonstrating exceptional performance across various tasks. However, substantial computational requirements make their deployment challenging on devices with limited resources. Recently, compression methods using low-rank matrix techniques have shown promise, yet these often lead to degraded accuracy or introduce significant overhead in parameters and inference latency. This paper introduces \textbf{Mo}dular \textbf{De}composition (MoDeGPT), a novel structured compression framework that does not need recovery fine-tuning while resolving the above drawbacks. MoDeGPT partitions the Transformer block into modules comprised of matrix pairs and reduces the hidden dimensions via reconstructing the module-level outputs. MoDeGPT is developed based on a theoretical framework that utilizes three well-established matrix decomposition algorithms -- Nystr\"om approximation, CR decomposition, and SVD -- and applies them to our redefined transformer modules. Our comprehensive experiments show MoDeGPT, without backward propagation, matches or surpasses previous structured compression methods that rely on gradient information, and saves 98% of compute costs on compressing a 13B model. On \textsc{Llama}-2/3 and OPT models, MoDeGPT maintains 90-95% zero-shot performance with 25-30% compression rates. Moreover, the compression can be done on a single GPU within a few hours and increases the inference throughput by up to 46%.

Related papers

• Language Models as Zero-shot Lossless Gradient Compressors: Towards General Neural Parameter Prior Models [66.1595537904019]
  Large language models (LLMs) can act as gradient priors in a zero-shot setting. We introduce LM-GC, a novel method that integrates LLMs with arithmetic coding.
  arXiv  Detail & Related papers  (2024-09-26T13:38:33Z)
• Mixture-of-Modules: Reinventing Transformers as Dynamic Assemblies of Modules [96.21649779507831]
  We propose a novel architecture dubbed mixture-of-modules (MoM) MoM is motivated by an intuition that any layer, regardless of its position, can be used to compute a token. We show that MoM provides not only a unified framework for Transformers but also a flexible and learnable approach for reducing redundancy.
  arXiv  Detail & Related papers  (2024-07-09T08:50:18Z)
• Demystifying the Compression of Mixture-of-Experts Through a Unified Framework [17.436189502801163]
  The Mixture of Experts (MoE) approach addresses this by dynamically selecting and activating only a subset of experts. MoE introduces potential redundancy (e.g., parameters) and extra costs (e.g., communication overhead) We first bridge this gap with a cutting-edge unified framework that not only seamlessly integrates mainstream compression methods but also helps systematically understand MoE compression.
  arXiv  Detail & Related papers  (2024-06-04T17:18:40Z)
• A Survey on Transformer Compression [84.18094368700379]
  Transformer plays a vital role in the realms of natural language processing (NLP) and computer vision (CV) Model compression methods reduce the memory and computational cost of Transformer. This survey provides a comprehensive review of recent compression methods, with a specific focus on their application to Transformer-based models.
  arXiv  Detail & Related papers  (2024-02-05T12:16:28Z)
• Rethinking Compression: Reduced Order Modelling of Latent Features in Large Language Models [9.91972450276408]
  This paper introduces an innovative approach for the parametric and practical compression of Large Language Models (LLMs) based on reduced order modelling. Our method represents a significant advancement in model compression by leveraging matrix decomposition, demonstrating superior efficacy compared to the prevailing state-of-the-art structured pruning method.
  arXiv  Detail & Related papers  (2023-12-12T07:56:57Z)
• TensorGPT: Efficient Compression of Large Language Models based on Tensor-Train Decomposition [19.897367559948336]
  We propose a training-free model compression approach based on the Matrix-Train Decomposition (TTD) We then investigate the low-rank structures extracted by this approach, in terms of the compression ratio, the language task performance, and latency on a typical low-end device (i.e. Raspberry Pi)
  arXiv  Detail & Related papers  (2023-07-02T09:33:09Z)
• Blockwise Compression of Transformer-based Models without Retraining [6.118476907408718]
  We propose BCT, a framework of blockwise compression for transformers without retraining. Unlike layerwise compression methods, BCT achieves finer compression of the entire transformer by operating blockwise. BCT effectively compresses all components of the model, including but not limited to the embedding, matrix multiplication, GELU, Softmax, layer normalization, and intermediate results.
  arXiv  Detail & Related papers  (2023-04-04T02:55:40Z)
• Compression of Generative Pre-trained Language Models via Quantization [62.80110048377957]
  We find that previous quantization methods fail on generative tasks due to the textithomogeneous word embeddings We propose a token-level contrastive distillation to learn distinguishable word embeddings, and a module-wise dynamic scaling to make quantizers adaptive to different modules.
  arXiv  Detail & Related papers  (2022-03-21T02:11:35Z)
• Efficient Micro-Structured Weight Unification and Pruning for Neural Network Compression [56.83861738731913]
  Deep Neural Network (DNN) models are essential for practical applications, especially for resource limited devices. Previous unstructured or structured weight pruning methods can hardly truly accelerate inference. We propose a generalized weight unification framework at a hardware compatible micro-structured level to achieve high amount of compression and acceleration.
  arXiv  Detail & Related papers  (2021-06-15T17:22:59Z)
• Spatiotemporal Entropy Model is All You Need for Learned Video Compression [9.227865598115024]
  We propose a framework to compress raw-pixel frames (rather than residual images) An entropy model is used to estimate thetemporal redundancy in a latent space rather than pixel level. Experiments showed that the proposed method outperforms state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2021-04-13T10:38:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.