Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods

• URL: http://arxiv.org/abs/2402.02834v2
• Date: Sun, 23 Jun 2024 08:45:33 GMT
• Title: Shortened LLaMA: Depth Pruning for Large Language Models with Comparison of Retraining Methods
• Authors: Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, Hyoung-Kyu Song,
• Abstract summary: We show that simple depth pruning can effectively compress large language models (LLMs) Our pruning method boosts inference speeds, especially under memory-constrained conditions. We hope this work can help build compact yet capable LLMs.
• Score: 5.135352292810664
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Structured pruning of modern large language models (LLMs) has emerged as a way of decreasing their high computational needs. Width pruning reduces the size of projection weight matrices (e.g., by removing attention heads) while maintaining the number of layers. Depth pruning, in contrast, removes entire layers or blocks, while keeping the size of the remaining weights unchanged. Most current research focuses on either width-only or a blend of width and depth pruning, with little comparative analysis between the two units (width vs. depth) concerning their impact on LLM inference efficiency. In this work, we show that simple depth pruning can effectively compress LLMs while achieving comparable or superior performance to recent width pruning studies. Our pruning method boosts inference speeds, especially under memory-constrained conditions that require limited batch sizes for running LLMs, where width pruning is ineffective. In retraining pruned models for quality recovery, continued pretraining on a large corpus markedly outperforms LoRA-based tuning, particularly at severe pruning ratios. We hope this work can help build compact yet capable LLMs. Code and models can be found at: https://github.com/Nota-NetsPresso/shortened-llm

Related papers

• BlockPruner: Fine-grained Pruning for Large Language Models [23.523314522663455]
  Research indicates certain layers in large language models (LLMs) harbor substantial redundancy, and pruning these layers has minimal impact on the overall performance. We propose a novel, training-free structured pruning approach called BlockPruner. We show that BlockPruner achieves more granular and effective pruning compared to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-06-15T11:03:33Z)
• Optimization-based Structural Pruning for Large Language Models without Back-Propagation [57.9629676017527]
  We propose an optimization-based structural pruning on Large-Language Models (LLMs) Our method learns the pruning masks in a probabilistic space directly by optimizing the loss of the pruned model. Our method operates for 2.7 hours with around 35GB memory for the 13B models on a single A100 GPU, and our pruned models outperform the state-of-the-arts w.r.t. perplexity.
  arXiv  Detail & Related papers  (2024-06-15T09:31:03Z)
• Fast and Effective Weight Update for Pruned Large Language Models [0.0]
  Pruning large language models (LLMs) is a challenging task due to their enormous size. Recent approaches have either ignored fine-tuning entirely, or attempted layer-wise weight updates. We propose a fast and effective weight update algorithm for pruned layers based on the Alternating Direction Method of Multipliers.
  arXiv  Detail & Related papers  (2024-01-01T23:10:23Z)
• Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs [67.38165028487242]
  We introduce Dynamic Sparse No Training (DSnoT), a training-free fine-tuning approach to fine-tune large language models (LLMs) Inspired by the Dynamic Sparse Training, DSnoT minimizes the reconstruction error between the dense and sparse LLMs. Our paper offers fresh insights into how to fine-tune sparse LLMs in an efficient training-free manner and open new venues to scale the great potential of sparsity to LLMs.
  arXiv  Detail & Related papers  (2023-10-13T07:38:52Z)
• Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruning [52.29522018586365]
  We study structured pruning as an effective means to develop smaller LLMs from pre-trained, larger models. Our approach employs two key techniques: (1) targeted structured pruning, which prunes a larger model to a specified target shape by removing layers, heads, and intermediate and hidden dimensions in an end-to-end manner, and (2) dynamic batch loading, which dynamically updates the composition of sampled data in each training batch based on varying losses across different domains.
  arXiv  Detail & Related papers  (2023-10-10T15:13:30Z)
• A Simple and Effective Pruning Approach for Large Language Models [58.716255689941896]
  Large Languages Models (LLMs) are natural candidates for network pruning methods. Existing methods, however, require either retraining, or solving a weight reconstruction problem reliant on second-order information. We introduce a novel, straightforward yet effective pruning method, termed Wanda (Pruning by Weights and activations), designed to induce sparsity in pretrained LLMs.
  arXiv  Detail & Related papers  (2023-06-20T17:18:20Z)
• LoRAPrune: Pruning Meets Low-Rank Parameter-Efficient Fine-Tuning [56.88751562302793]
  Low-rank adaption (LoRA) has emerged to fine-tune large language models (LLMs) LoRAPrune is a new framework that delivers an accurate structured pruned model in a highly memory-efficient manner. LoRAPrune achieves a reduction in perplexity by 4.81 on WikiText2 and 3.46 on PTB, while also decreasing memory usage by 52.6%.
  arXiv  Detail & Related papers  (2023-05-28T15:15:48Z)
• LLM-Pruner: On the Structural Pruning of Large Language Models [65.02607075556742]
  Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. We tackle the compression of LLMs within the bound of two constraints: being task-agnostic and minimizing the reliance on the original training dataset. Our method, named LLM-Pruner, adopts structural pruning that selectively removes non-critical coupled structures.
  arXiv  Detail & Related papers  (2023-05-19T12:10:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.