ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models

• URL: http://arxiv.org/abs/2406.16635v2
• Date: Thu, 17 Oct 2024 15:45:10 GMT
• Title: ShadowLLM: Predictor-based Contextual Sparsity for Large Language Models
• Authors: Yash Akhauri, Ahmed F AbouElhamayed, Jordan Dotzel, Zhiru Zhang, Alexander M Rush, Safeen Huda, Mohamed S Abdelfattah,
• Abstract summary: We develop a novel predictor called ShadowLLM, which can shadow the LLM behavior and enforce better sparsity patterns. ShadowLLM achieves up to a 20% speed-up over the state-of-the-art DejaVu framework.
• Score: 67.97667465509504
• License:
• Abstract: The high power consumption and latency-sensitive deployments of large language models (LLMs) have motivated efficiency techniques like quantization and sparsity. Contextual sparsity, where the sparsity pattern is input-dependent, is crucial in LLMs because the permanent removal of attention heads or neurons from LLMs can significantly degrade accuracy. Prior work has attempted to model contextual sparsity using neural networks trained to predict activation magnitudes, which can be used to dynamically prune structures with low predicted activation magnitude. In this paper, we look beyond magnitude-based pruning criteria to assess attention head and neuron importance in LLMs. We develop a novel predictor called ShadowLLM, which can shadow the LLM behavior and enforce better sparsity patterns, resulting in over 15% improvement in end-to-end accuracy compared to prior methods. In addition, ShadowLLM achieves up to a 20% speed-up over the state-of-the-art DejaVu framework. These enhancements are validated on Llama-2 and OPT models with up to 30 billion parameters. Our code is available at \href{https://github.com/abdelfattah-lab/shadow_llm/}{ShadowLLM}.

Related papers

• Sparsing Law: Towards Large Language Models with Greater Activation Sparsity [62.09617609556697]
  Activation sparsity denotes the existence of substantial weakly-contributed elements within activation outputs that can be eliminated. We propose PPL-$p%$ sparsity, a precise and performance-aware activation sparsity metric. We show that ReLU is more efficient as the activation function than SiLU and can leverage more training data to improve activation sparsity.
  arXiv  Detail & Related papers  (2024-11-04T17:59:04Z)
• Get Confused Cautiously: Textual Sequence Memorization Erasure with Selective Entropy Maximization [17.20276556057748]
  Large Language Models (LLMs) have been found to memorize and recite some of the textual sequences from their training set verbatim. This Textual Sequence Memorization (TSM) phenomenon leads to a high demand to regulate LLM output to prevent it from generating certain memorized text. Existing methods for TSM erasure fail to forget massive memorized samples without substantially jeopardizing the model utility.
  arXiv  Detail & Related papers  (2024-08-09T10:26:11Z)
• Achieving Sparse Activation in Small Language Models [9.05326883263473]
  Sparse activation is a useful technique to reduce the computing cost of Large Language Models (LLMs) without retraining or adaptation efforts. In this paper, we aim to achieve sparse activation in Small Language Models (SLMs) We first show that the existing sparse activation schemes in LLMs that build on neurons' output magnitudes cannot be applied to SLMs, and activating neurons based on their attribution scores is a better alternative.
  arXiv  Detail & Related papers  (2024-06-03T03:21:49Z)
• Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment [56.44025052765861]
  Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs. We show a total speedup on CPUs for sparse-quantized LLaMA models of up to 8.6x.
  arXiv  Detail & Related papers  (2024-05-06T16:03:32Z)
• Learn To be Efficient: Build Structured Sparsity in Large Language Models [17.940183066850565]
  Large Language Models (LLMs) have achieved remarkable success with their billion-level parameters, yet they incur high inference overheads. Existing methods only focus on utilizing this naturally formed activation sparsity in a post-training setting. We introduce a novel training algorithm, Learn-To-be-Efficient (LTE), designed to train efficiency-aware LLMs.
  arXiv  Detail & Related papers  (2024-02-09T01:18:16Z)
• ReLU$^2$ Wins: Discovering Efficient Activation Functions for Sparse LLMs [91.31204876440765]
  We introduce a general method that defines neuron activation through neuron output magnitudes and a tailored magnitude threshold. To find the most efficient activation function for sparse computation, we propose a systematic framework. We conduct thorough experiments on LLMs utilizing different activation functions, including ReLU, SwiGLU, ReGLU, and ReLU$2$.
  arXiv  Detail & Related papers  (2024-02-06T08:45:51Z)
• One-Shot Sensitivity-Aware Mixed Sparsity Pruning for Large Language Models [42.95555008229016]
  We propose a method based on Hessian sensitivity-aware mixed sparsity pruning to prune LLMs to at least 50% sparsity without the need of any retraining. The advantages of the proposed method exhibit even more when the sparsity is extremely high.
  arXiv  Detail & Related papers  (2023-10-14T05:43:09Z)
• Dynamic Sparse No Training: Training-Free Fine-tuning for Sparse LLMs [67.38165028487242]
  We introduce Dynamic Sparse No Training (DSnoT), a training-free fine-tuning approach to fine-tune large language models (LLMs) Inspired by the Dynamic Sparse Training, DSnoT minimizes the reconstruction error between the dense and sparse LLMs. Our paper offers fresh insights into how to fine-tune sparse LLMs in an efficient training-free manner and open new venues to scale the great potential of sparsity to LLMs.
  arXiv  Detail & Related papers  (2023-10-13T07:38:52Z)
• To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis [50.31589712761807]
  Large language models (LLMs) are notoriously token-hungry during pre-training, and high-quality text data on the web is approaching its scaling limit for LLMs. We investigate the consequences of repeating pre-training data, revealing that the model is susceptible to overfitting. Second, we examine the key factors contributing to multi-epoch degradation, finding that significant factors include dataset size, model parameters, and training objectives.
  arXiv  Detail & Related papers  (2023-05-22T17:02:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.