DReSS: Data-driven Regularized Structured Streamlining for Large Language Models

• URL: http://arxiv.org/abs/2501.17905v2
• Date: Mon, 10 Feb 2025 04:07:04 GMT
• Title: DReSS: Data-driven Regularized Structured Streamlining for Large Language Models
• Authors: Mingkuan Feng, Jinyang Wu, Shuai Zhang, Pengpeng Shao, Ruihan Jin, Zhengqi Wen, Jianhua Tao, Feihu Che,
• Abstract summary: Large language models (LLMs) have achieved significant progress across various domains, but their increasing scale results in high computational and memory costs. We propose a novel paradigm that first applies regularization, then prunes, and finally finetunes. By leveraging a small amount of data to regularize the components to be pruned, DReSS explicitly transfers the important information to the remaining parts of the model in advance.
• Score: 30.47317140878219
• License:
• Abstract: Large language models (LLMs) have achieved significant progress across various domains, but their increasing scale results in high computational and memory costs. Recent studies have revealed that LLMs exhibit sparsity, providing the potential to reduce model size through pruning techniques. However, existing pruning methods typically follow a prune-then-finetune paradigm. Since the pruned components still contain valuable information, their direct removal often leads to irreversible performance degradation, imposing a substantial computational burden to recover performance during finetuning. In this paper, we propose a novel paradigm that first applies regularization, then prunes, and finally finetunes. Based on this paradigm, we introduce DReSS, a simple and effective Data-driven Regularized Structured Streamlining method for LLMs. By leveraging a small amount of data to regularize the components to be pruned, DReSS explicitly transfers the important information to the remaining parts of the model in advance. Compared to direct pruning, this can reduce the information loss caused by parameter removal, thereby enhancing its language modeling capabilities. Experimental results demonstrate that DReSS significantly outperforms existing pruning methods even under extreme pruning ratios, significantly reducing latency and increasing throughput.

Related papers

• PASER: Post-Training Data Selection for Efficient Pruned Large Language Model Recovery [11.20326903218271]
  Post-training techniques such as instruction tuning are commonly employed to recover model performance. However, some instruction data irrelevant to model capability recovery may introduce negative effects. We propose PASER to identify instructions where model capabilities are most severely compromised.
  arXiv  Detail & Related papers  (2025-02-18T07:11:08Z)
• Lightweight and Post-Training Structured Pruning for On-Device Large Lanaguage Models [11.93284417365518]
  We introduce COMP, a lightweight post-training structured pruning method that employs a hybrid-granularity pruning strategy. COMP improves performance by 6.13% on the LLaMA-2-7B model with a 20% pruning ratio compared to LLM-Pruner.
  arXiv  Detail & Related papers  (2025-01-25T16:03:58Z)
• DRPruning: Efficient Large Language Model Pruning through Distributionally Robust Optimization [61.492590008258986]
  Large language models (LLMs) deliver impressive results but face challenges from increasing model sizes and computational costs. We propose DRPruning, which incorporates distributionally robust optimization to restore balanced performance across domains.
  arXiv  Detail & Related papers  (2024-11-21T12:02:39Z)
• Enhancing One-shot Pruned Pre-trained Language Models through Sparse-Dense-Sparse Mechanism [25.36736897890854]
  Pre-trained language models (PLMs) are engineered to be robust in contextual understanding and exhibit outstanding performance in various natural language processing tasks. Modern pruning strategies employ one-shot techniques to compress PLMs without the need for retraining on task-specific or otherwise general data. We propose SDS, a Sparse-Dense-Sparse pruning framework to enhance the performance of the pruned PLMs from a weight distribution optimization perspective.
  arXiv  Detail & Related papers  (2024-08-20T01:05:45Z)
• Greedy Output Approximation: Towards Efficient Structured Pruning for LLMs Without Retraining [16.026565606764954]
  We simplify the pruning process for Transformer-based large language models (LLMs) We propose two inference-aware pruning criteria derived from the optimization perspective of output approximation. We also introduce a two-step reconstruction technique to mitigate pruning errors without model retraining.
  arXiv  Detail & Related papers  (2024-07-26T23:53:59Z)
• Straightforward Layer-wise Pruning for More Efficient Visual Adaptation [0.0]
  We propose a Straightforward layer-wise pruning method, called SLS, for pruning PETL-transferred models. Our study reveals that layer-wise pruning, with a focus on storing pruning indices, addresses storage volume concerns.
  arXiv  Detail & Related papers  (2024-07-19T14:10:35Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• Low-rank finetuning for LLMs: A fairness perspective [54.13240282850982]
  Low-rank approximation techniques have become the de facto standard for fine-tuning Large Language Models. This paper investigates the effectiveness of these methods in capturing the shift of fine-tuning datasets from the initial pre-trained data distribution. We show that low-rank fine-tuning inadvertently preserves undesirable biases and toxic behaviors.
  arXiv  Detail & Related papers  (2024-05-28T20:43:53Z)
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
  Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
  arXiv  Detail & Related papers  (2023-05-22T17:02:15Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)
• Movement Pruning: Adaptive Sparsity by Fine-Tuning [115.91907953454034]
  Magnitude pruning is a widely used strategy for reducing model size in pure supervised learning. We propose the use of movement pruning, a simple, deterministic first-order weight pruning method. Experiments show that when pruning large pretrained language models, movement pruning shows significant improvements in high-sparsity regimes.
  arXiv  Detail & Related papers  (2020-05-15T17:54:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.