decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points

• URL: http://arxiv.org/abs/2404.12759v1
• Date: Fri, 19 Apr 2024 10:02:53 GMT
• Title: decoupleQ: Towards 2-bit Post-Training Uniform Quantization via decoupling Parameters into Integer and Floating Points
• Authors: Yi Guo, Fanliu Kong, Xiaoyang Li, Hui Li, Wei Chen, Xiaogang Tian, Jinping Cai, Yang Zhang, Shouda Liu,
• Abstract summary: decoupleQ abandons the traditional quantization paradigm and decouples the model parameters into integer and floating-point parts. Our method has achieved well on-line accuracy near fp16/bf16 on the 2-bit quantization of large speech models in ByteDance.
• Score: 10.238677144792279
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Quantization emerges as one of the most promising compression technologies for deploying efficient large models for various real time application in recent years. Considering that the storage and IO of weights take up the vast majority of the overhead inside a large model, weight only quantization can lead to large gains. However, existing quantization schemes suffer from significant accuracy degradation at very low bits, or require some additional computational overhead when deployed, making it difficult to be applied to large-scale applications in industry. In this paper, we propose decoupleQ, achieving a substantial increase in model accuracy, especially at very low bits. decoupleQ abandons the traditional heuristic quantization paradigm and decouples the model parameters into integer and floating-point parts, thus transforming the quantization problem into a traditional mathematical optimization problem with constraints, which is then solved alternatively by off-the-shelf optimization methods. Quantization via decoupleQ is linear and uniform, making it hardware-friendlier than non-uniform counterpart, and enabling the idea to be migrated to high-bit quantization to enhance its robustness. Our method has achieved well on-line accuracy near fp16/bf16 on the 2-bit quantization of large speech models in ByteDance. The code is available at https://github.com/bytedance/decoupleQ

Related papers

• LeanQuant: Accurate Large Language Model Quantization with Loss-Error-Aware Grid [36.33062038680275]
  Large language models (LLMs) have numerous applications across various domains. Weight quantization is an effective technique for reducing the decoding latency and memory requirements of LLMs. We propose LeanQuant, which learns a loss-error-aware quantization grid by leveraging the inverse diagonal Hessian.
  arXiv  Detail & Related papers  (2024-07-14T00:23:51Z)
• EfficientQAT: Efficient Quantization-Aware Training for Large Language Models [62.904403513409484]
  Large language models (LLMs) are integral to modern natural language processing and artificial intelligence. We propose Efficient Quantization-Aware Training (EfficientQAT), a novel quantization technique for compressing LLMs. Extensive experiments demonstrate that EfficientQAT outperforms previous quantization methods across a range of models.
  arXiv  Detail & Related papers  (2024-07-10T17:53:30Z)
• FrameQuant: Flexible Low-Bit Quantization for Transformers [27.93241211038938]
  Post-Training Quantization seeks to modify a pre-trained model and quantize it to eight bits or lower. We show that (almost) two-bit quantization for Transformer models promises sizable efficiency gains.
  arXiv  Detail & Related papers  (2024-03-10T04:01:49Z)
• Post-Training Quantization for Re-parameterization via Coarse & Fine Weight Splitting [13.270381125055275]
  We propose a coarse & fine weight splitting (CFWS) method to reduce quantization error of weight. We develop an improved KL metric to determine optimal quantization scales for activation. For example, the quantized RepVGG-A1 model exhibits a mere 0.3% accuracy loss.
  arXiv  Detail & Related papers  (2023-12-17T02:31:20Z)
• QUIK: Towards End-to-End 4-Bit Inference on Generative Large Language Models [57.04178959678024]
  We show that the majority of inference computations for large generative models can be performed with both weights and activations being cast to 4 bits. We achieve this via a hybrid quantization strategy called QUIK, which compresses most of the weights and activations to 4-bit. We provide GPU kernels matching the QUIK format with highly-efficient layer-wise runtimes, which lead to practical end-to-end throughput improvements of up to 3.4x.
  arXiv  Detail & Related papers  (2023-10-13T17:15:05Z)
• NUPES : Non-Uniform Post-Training Quantization via Power Exponent Search [7.971065005161565]
  quantization is a technique to convert floating point representations to low bit-width fixed point representations. We show how to learn new quantized weights over the entire quantized space. We show the ability of the method to achieve state-of-the-art compression rates in both, data-free and data-driven configurations.
  arXiv  Detail & Related papers  (2023-08-10T14:19:58Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)
• DeepGEMM: Accelerated Ultra Low-Precision Inference on CPU Architectures using Lookup Tables [49.965024476651706]
  DeepGEMM is a lookup table based approach for the execution of ultra low-precision convolutional neural networks on SIMD hardware. Our implementation outperforms corresponding 8-bit integer kernels by up to 1.74x on x86 platforms.
  arXiv  Detail & Related papers  (2023-04-18T15:13:10Z)
• Q-ASR: Integer-only Zero-shot Quantization for Efficient Speech Recognition [65.7040645560855]
  We propose Q-ASR, an integer-only, zero-shot quantization scheme for ASR models. We show negligible WER change as compared to the full-precision baseline models. Q-ASR exhibits a large compression rate of more than 4x with small WER degradation.
  arXiv  Detail & Related papers  (2021-03-31T06:05:40Z)
• Once Quantization-Aware Training: High Performance Extremely Low-bit Architecture Search [112.05977301976613]
  We propose to combine Network Architecture Search methods with quantization to enjoy the merits of the two sides. We first propose the joint training of architecture and quantization with a shared step size to acquire a large number of quantized models. Then a bit-inheritance scheme is introduced to transfer the quantized models to the lower bit, which further reduces the time cost and improves the quantization accuracy.
  arXiv  Detail & Related papers  (2020-10-09T03:52:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.