QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models

• URL: http://arxiv.org/abs/2307.03738v1
• Date: Fri, 7 Jul 2023 17:46:08 GMT
• Title: QIGen: Generating Efficient Kernels for Quantized Inference on Large Language Models
• Authors: Tommaso Pegolotti, Elias Frantar, Dan Alistarh, Markus P\"uschel
• Abstract summary: We present an automatic code generation approach for supporting quantized generative inference on LLMs such as LLaMA or OPT on off-the-shelf CPUs. Results on CPU-based inference for LLaMA models show that our approach can lead to high performance and high accuracy, comparing favorably to the best existing open-source solution.
• Score: 22.055655390093722
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present ongoing work on a new automatic code generation approach for supporting quantized generative inference on LLMs such as LLaMA or OPT on off-the-shelf CPUs. Our approach is informed by the target architecture and a performance model, including both hardware characteristics and method-specific accuracy constraints. Results on CPU-based inference for LLaMA models show that our approach can lead to high performance and high accuracy, comparing favorably to the best existing open-source solution. A preliminary implementation is available at https://github.com/IST-DASLab/QIGen.

Related papers

• LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators [1.1028525384019312]
  Large Language Models (LLMs) have propelled groundbreaking advancements across several domains and are commonly used for text generation applications. We introduce LLM-Inference-Bench, a comprehensive benchmarking suite to evaluate the hardware inference performance of LLMs. Our benchmarking results reveal the strengths and limitations of various models, hardware platforms, and inference frameworks.
  arXiv  Detail & Related papers  (2024-10-31T18:34:59Z)
• COrAL: Order-Agnostic Language Modeling for Efficient Iterative Refinement [80.18490952057125]
  Iterative refinement has emerged as an effective paradigm for enhancing the capabilities of large language models (LLMs) on complex tasks. We propose Context-Wise Order-Agnostic Language Modeling (COrAL) to overcome these challenges. Our approach models multiple token dependencies within manageable context windows, enabling the model to perform iterative refinement internally.
  arXiv  Detail & Related papers  (2024-10-12T23:56:19Z)
• Search for Efficient Large Language Models [52.98684997131108]
  Large Language Models (LLMs) have long held sway in the realms of artificial intelligence research. Weight pruning, quantization, and distillation have been embraced to compress LLMs, targeting memory reduction and inference acceleration. Most model compression techniques concentrate on weight optimization, overlooking the exploration of optimal architectures.
  arXiv  Detail & Related papers  (2024-09-25T21:32:12Z)
• Inference Performance Optimization for Large Language Models on CPUs [4.7230692120532485]
  Large language models (LLMs) have shown exceptional performance and vast potential across diverse tasks. When GPU hardware resources are limited, we can explore alternative options on CPUs. In this paper, we introduce an easily deployable inference performance optimization solution aimed at accelerating LLMs on CPUs.
  arXiv  Detail & Related papers  (2024-07-10T01:53:49Z)
• Tender: Accelerating Large Language Models via Tensor Decomposition and Runtime Requantization [0.6445087473595953]
  Large language models (LLMs) demonstrate outstanding performance in various tasks in machine learning. deploying LLM inference poses challenges due to the high compute and memory requirements. We present Tender, an algorithm-hardware co-design solution that enables efficient deployment of LLM inference at low precision.
  arXiv  Detail & Related papers  (2024-06-16T09:51:55Z)
• Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment [56.44025052765861]
  Large language models (LLMs) have revolutionized Natural Language Processing (NLP), but their size creates computational bottlenecks. We introduce a novel approach to create accurate, sparse foundational versions of performant LLMs. We show a total speedup on CPUs for sparse-quantized LLaMA models of up to 8.6x.
  arXiv  Detail & Related papers  (2024-05-06T16:03:32Z)
• Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models [90.14693869269519]
  MoE LLMs can achieve higher performance with fewer parameters, but it is still hard to deploy them due to their immense parameter sizes. This paper mainly aims to enhance the deployment efficiency of MoE LLMs by introducing plug-and-play expert-level sparsification techniques.
  arXiv  Detail & Related papers  (2024-02-22T18:56:07Z)
• Efficient LLM Inference on CPUs [8.802223672775844]
  Large language models (LLMs) have demonstrated remarkable performance and tremendous potential across a wide range of tasks. deploying these models has been challenging due to the astronomical amount of model parameters. We propose an effective approach that can make the deployment of LLMs more efficiently.
  arXiv  Detail & Related papers  (2023-11-01T13:08:50Z)
• The Synergy of Speculative Decoding and Batching in Serving Large Language Models [3.3849225405083336]
  We propose a new speculative decoding strategy that chooses the optimal speculation length for different batch sizes. Our evaluations show that our proposed method can achieve equal or better performance than the state-of-the-art speculation decoding schemes with fixed speculation length.
  arXiv  Detail & Related papers  (2023-10-28T20:36:36Z)
• Cheaply Evaluating Inference Efficiency Metrics for Autoregressive Transformer APIs [66.30706841821123]
  Large language models (LLMs) power many state-of-the-art systems in natural language processing. LLMs are extremely computationally expensive, even at inference time. We propose a new metric for comparing inference efficiency across models.
  arXiv  Detail & Related papers  (2023-05-03T21:51:42Z)
• Energy-efficient Task Adaptation for NLP Edge Inference Leveraging Heterogeneous Memory Architectures [68.91874045918112]
  adapter-ALBERT is an efficient model optimization for maximal data reuse across different tasks. We demonstrate the advantage of mapping the model to a heterogeneous on-chip memory architecture by performing simulations on a validated NLP edge accelerator.
  arXiv  Detail & Related papers  (2023-03-25T14:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.