Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring

• URL: http://arxiv.org/abs/2501.13331v2
• Date: Wed, 05 Feb 2025 08:10:45 GMT
• Title: Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring
• Authors: Dongyoung Lee, Seungkyu Choi, Ik Joon Chang,
• Abstract summary: We propose QRazor, a simple yet effective quantization scheme that enables 4-bit quantization of weights, activations, and KV cache in transformer-based language models. QRazor operates in two stages: first, quantizing data using 8 or 16-bit integers as a basis with absolute max scaling to preserve accuracy close to full-precision models, and second, compressing the quantized data to 4-bit using our significant data razoring (SDR) technique.
• Score: 2.983583925806601
• License:
• Abstract: Large-scale language models (LLMs) excel in language processing tasks but face deployment challenges due to high memory and computational demands. While low-bit quantization, such as 4-bit techniques, offers a potential solution, these methods often suffer from significant accuracy loss or require considerable effort for implementation such as reordering, rotation, etc. To address these challenges, we propose QRazor, a simple yet effective quantization scheme that enables 4-bit quantization of weights, activations, and KV cache in transformer-based LLMs. QRazor operates in two stages: first, quantizing data using 8 or 16-bit integers as a basis with absolute max scaling to preserve accuracy close to full-precision models, and second, compressing the quantized data to 4-bit using our significant data razoring (SDR) technique, which retains only the four most salient bits. Without any additional requirment of fine-tuning or additional training, QRazor achieves performance similar or better compared to state-of-the-art in 4-bit quantization method, surpassing Smoothquant and QLLM by over 12 points and Quarot(RTN) by more than 2.9 points in zero-shot reasoning task accuracy on the LLaMA2-7B model. Additionally, we introduce an integer-based arithmetic unit optimized for QRazor, allowing direct low-precision operations on SDR data without decompression.

Related papers

• ParetoQ: Scaling Laws in Extremely Low-bit LLM Quantization [58.84018707089315]
  We present a unified framework for rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings. We show that ternary, 2-bit, and 3-bit quantization maintains comparable performance in the size-accuracy trade-off. Considering hardware constraints, 2-bit quantization offers promising potential for memory reduction and speedup.
  arXiv  Detail & Related papers  (2025-02-04T18:59:26Z)
• ResQ: Mixed-Precision Quantization of Large Language Models with Low-Rank Residuals [10.860081994662645]
  Post-training quantization of large language models (LLMs) holds the promise in reducing the prohibitive computational cost at inference time. We propose ResQ, a PTQ method that pushes further the state-of-the-art. We demonstrate that ResQ outperforms recent uniform and mixed precision PTQ methods on a variety of benchmarks.
  arXiv  Detail & Related papers  (2024-12-18T22:01:55Z)
• AMXFP4: Taming Activation Outliers with Asymmetric Microscaling Floating-Point for 4-bit LLM Inference [6.699442219974261]
  We propose Asymmetric Microscaling 4-bit Floating-Point (AMXFP4) for efficient Large Language Models inference. Unlike conventional 4-bit quantization methods that rely on data rotation and costly calibration, AMXFP4 uses asymmetric shared scales for direct 4-bit casting. Our AMXFP4 format significantly outperforms MXFP4 and other leading quantization techniques, enabling robust, calibration-free 4-bit inference.
  arXiv  Detail & Related papers  (2024-11-15T03:11:19Z)
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models [67.67135738642547]
  Post-training quantization (PTQ) is a powerful compression technique investigated in large language models (LLMs) Existing PTQ methods are not ideal in terms of accuracy and efficiency, especially with below 4 bit-widths. This paper presents a Salience-Driven Mixed-Precision Quantization scheme for LLMs, namely SliM-LLM.
  arXiv  Detail & Related papers  (2024-05-23T16:21:48Z)
• Enhancing Computation Efficiency in Large Language Models through Weight and Activation Quantization [12.655230451207956]
  This paper focuses on post-training quantization (PTQ) in Large Language Models (LLMs) We present two innovative techniques: activation-quantization-aware scaling (AQAS) and sequence-length-aware calibration (SLAC) We demonstrate that our techniques significantly boost task accuracies to levels comparable with full-precision models.
  arXiv  Detail & Related papers  (2023-11-09T06:19:51Z)
• QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models [57.04178959678024]
  We show that the majority of inference computations for large generative models can be performed with both weights and activations being cast to 4 bits. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit. We provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x.
  arXiv  Detail & Related papers  (2023-10-13T17:15:05Z)
• Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs [16.596819845726625]
  SignRound is a method that leverages signed gradient descent (SignSGD) to optimize rounding values and weight clipping in just 200 steps. It delivers exceptional results across 2 to 4 bits while minimizing tuning costs and avoiding additional inference overhead. It also demonstrates strong generalization in recent models, achieving near-lossless 4-bit quantization in most scenarios.
  arXiv  Detail & Related papers  (2023-09-11T14:58:23Z)
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models [57.27101446992148]
  Large language models (LLMs) have revolutionized natural language processing tasks. Recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM. We introduce an Omnidirectionally calibrated Quantization technique for LLMs, which achieves good performance in diverse quantization settings.
  arXiv  Detail & Related papers  (2023-08-25T02:28:35Z)
• SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compression [76.73007709690306]
  We introduce the Sparse-Quantized Representation (SpQR), a new compressed format and quantization technique. SpQR achieves relative accuracy losses of less than 1% in perplexity for highly-accurate LLaMA and Falcon LLMs. This makes it possible to run 33B parameter LLM on a single 24 GB consumer GPU without any performance degradation at 15% speedup.
  arXiv  Detail & Related papers  (2023-06-05T17:53:28Z)
• Q-ASR: Integer-only Zero-shot Quantization for Efficient Speech Recognition [65.7040645560855]
  We propose Q-ASR, an integer-only, zero-shot quantization scheme for ASR models. We show negligible WER change as compared to the full-precision baseline models. Q-ASR exhibits a large compression rate of more than 4x with small WER degradation.
  arXiv  Detail & Related papers  (2021-03-31T06:05:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.