Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs

• URL: http://arxiv.org/abs/2309.05516v5
• Date: Tue, 08 Oct 2024 02:02:35 GMT
• Title: Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs
• Authors: Wenhua Cheng, Weiwei Zhang, Haihao Shen, Yiyang Cai, Xin He, Kaokao Lv, Yi Liu,
• Abstract summary: SignRound is a method that leverages signed gradient descent (SignSGD) to optimize rounding values and weight clipping in just 200 steps. It delivers exceptional results across 2 to 4 bits while minimizing tuning costs and avoiding additional inference overhead. It also demonstrates strong generalization in recent models, achieving near-lossless 4-bit quantization in most scenarios.
• Score: 16.596819845726625
• License:
• Abstract: Large Language Models (LLMs) have demonstrated exceptional proficiency in language-related tasks, but their deployment poses significant challenges due to substantial memory and storage requirements. Weight-only quantization has emerged as a promising solution, significantly reducing memory and storage needs without sacrificing too much performance. In this study, we introduce SignRound, a method that leverages signed gradient descent (SignSGD) to optimize rounding values and weight clipping in just 200 steps. SignRound integrates the advantages of Quantization-Aware Training (QAT) and Post-Training Quantization (PTQ), delivering exceptional results across 2 to 4 bits while minimizing tuning costs and avoiding additional inference overhead. For example, SignRound achieved absolute average accuracy improvements ranging from 6.91% to 33.22% at 2bits, as measured by the average zero-shot accuracy across 11 tasks. It also demonstrates strong generalization in recent models, achieving near-lossless 4-bit quantization in most scenarios. The source code is publicly available at https://github.com/intel/auto-round.

Related papers

• QuZO: Quantized Zeroth-Order Fine-Tuning for Large Language Models [27.730213115659986]
  Language Models (LLMs) are often quantized to lower precision to reduce the memory cost and latency in inference. Traditional fine-tuning methods require backpropagation, which are error-prone in the low-precision settings. We propose the Quantized Zeroth-Order (QuZO) framework, specifically designed for fine-tuning LLMs through low-precision forward passes.
  arXiv  Detail & Related papers  (2025-02-17T22:20:31Z)
• Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring [2.983583925806601]
  We propose QRazor, a simple yet effective quantization scheme that enables 4-bit quantization of weights, activations, and KV cache in transformer-based language models. QRazor operates in two stages: first, quantizing data using 8 or 16-bit integers as a basis with absolute max scaling to preserve accuracy close to full-precision models, and second, compressing the quantized data to 4-bit using our significant data razoring (SDR) technique.
  arXiv  Detail & Related papers  (2025-01-23T02:20:08Z)
• GWQ: Gradient-Aware Weight Quantization for Large Language Models [63.89099994367657]
  Large language models (LLMs) show impressive performance in solving complex language tasks. LLMs to low bits can enable them to run on resource-constrained devices, often leading to performance degradation. We propose gradient-aware weight quantization (GWQ), the first quantization approach for low-bit weight quantization.
  arXiv  Detail & Related papers  (2024-10-30T11:16:04Z)
• EfficientQAT: Efficient Quantization-Aware Training for Large Language Models [50.525259103219256]
  quantization-aware training (QAT) offers a solution by reducing memory consumption through low-bit representations with minimal accuracy loss. We propose Efficient Quantization-Aware Training (EfficientQAT), a more feasible QAT algorithm. EfficientQAT involves two consecutive phases: Block-wise training of all parameters (Block-AP) and end-to-end training of quantization parameters (E2E-QP)
  arXiv  Detail & Related papers  (2024-07-10T17:53:30Z)
• GPTQT: Quantize Large Language Models Twice to Push the Efficiency [1.3149617027696827]
  This paper introduces a new post-training quantization method, GPTQT, to reduce memory usage and enhance processing speed. Practice has shown that minimizing the quantization error of weights is ineffective, leading to overfitting. GPTQT employs a progressive two-step approach: initially quantizing weights using Linear quantization to a relatively high bit, followed by converting obtained int weight to lower bit binary coding.
  arXiv  Detail & Related papers  (2024-07-03T08:08:01Z)
• APTQ: Attention-aware Post-Training Mixed-Precision Quantization for Large Language Models [12.006605064782567]
  We propose APTQ (Attention-aware Post-Training Mixed-Precision Quantization) for Large Language Models. We leverage the Hessian trace as a sensitivity metric for mixed-precision quantization, ensuring an informed precision reduction. Experiments show APTQ surpasses previous quantization methods, achieving an average of 4 bit width a 5.22 perplexity.
  arXiv  Detail & Related papers  (2024-02-21T07:45:22Z)
• QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models [57.04178959678024]
  We show that the majority of inference computations for large generative models can be performed with both weights and activations being cast to 4 bits. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit. We provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x.
  arXiv  Detail & Related papers  (2023-10-13T17:15:05Z)
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models [57.27101446992148]
  Large language models (LLMs) have revolutionized natural language processing tasks. Recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM. We introduce an Omnidirectionally calibrated Quantization technique for LLMs, which achieves good performance in diverse quantization settings.
  arXiv  Detail & Related papers  (2023-08-25T02:28:35Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)
• Q-ASR: Integer-only Zero-shot Quantization for Efficient Speech Recognition [65.7040645560855]
  We propose Q-ASR, an integer-only, zero-shot quantization scheme for ASR models. We show negligible WER change as compared to the full-precision baseline models. Q-ASR exhibits a large compression rate of more than 4x with small WER degradation.
  arXiv  Detail & Related papers  (2021-03-31T06:05:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.