QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks

• URL: http://arxiv.org/abs/2402.04396v2
• Date: Tue, 4 Jun 2024 04:51:52 GMT
• Title: QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks
• Authors: Albert Tseng, Jerry Chee, Qingyao Sun, Volodymyr Kuleshov, Christopher De Sa,
• Abstract summary: Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing their weights to low-precision. We introduce QuIP#, a weight-only PTQ method that achieves state-of-the-art results in extreme compression regimes. Our experiments show that QuIP# outperforms existing PTQ methods, enables new behaviors in PTQ scaling, and supports fast inference.
• Score: 37.66253003964376
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Post-training quantization (PTQ) reduces the memory footprint of LLMs by quantizing their weights to low-precision. In this work, we introduce QuIP#, a weight-only PTQ method that achieves state-of-the-art results in extreme compression regimes ($\le$ 4 bits per weight) using three novel techniques. First, QuIP# improves QuIP's (Chee et al., 2023) incoherence processing by using the randomized Hadamard transform, which is faster and has better theoretical properties. Second, QuIP# uses vector quantization to take advantage of the ball-shaped sub-Gaussian distribution that incoherent weights possess: specifically, we introduce a set of hardware-efficient codebooks based on the highly symmetric $E_8$ lattice, which achieves the optimal 8-dimension unit ball packing. Third, QuIP# uses fine-tuning to improve fidelity to the original model. Our experiments show that QuIP# outperforms existing PTQ methods, enables new behaviors in PTQ scaling, and supports fast inference. Our code can be found at https://github.com/Cornell-RelaxML/quip-sharp.

Related papers

• Pyramid Vector Quantization for LLMs [8.779688608449902]
  Pyramid Vector Quantization (PVQ) for large language models. PVQ uses a fixed integer lattice on the sphere by projecting points onto the 1-sphere, which allows for efficient encoding and decoding without requiring an explicit codebook in memory. We achieve state-of-the-art quantization performance with pareto-optimal trade-off between performance and bits per weight and bits per activation, compared to compared methods.
  arXiv  Detail & Related papers  (2024-10-22T11:57:32Z)
• GPTQT: Quantize Large Language Models Twice to Push the Efficiency [1.3149617027696827]
  This paper introduces a new post-training quantization method, GPTQT, to reduce memory usage and enhance processing speed. Practice has shown that minimizing the quantization error of weights is ineffective, leading to overfitting. GPTQT employs a progressive two-step approach: initially quantizing weights using Linear quantization to a relatively high bit, followed by converting obtained int weight to lower bit binary coding.
  arXiv  Detail & Related papers  (2024-07-03T08:08:01Z)
• QTIP: Quantization with Trellises and Incoherence Processing [29.917017118524246]
  Post-training quantization (PTQ) reduces the memory footprint of LLMs. Recent state-of-the-art PTQ approaches use vector quantization (VQ) to quantize multiple weights at once. We introduce QTIP, which instead uses trellis coded quantization (TCQ) to achieve ultra-high-dimensional quantization.
  arXiv  Detail & Related papers  (2024-06-17T06:03:13Z)
• OAC: Output-adaptive Calibration for Accurate Post-training Quantization [30.115888331426515]
  Post-training Quantization (PTQ) techniques have been developed to compress Large Language Models (LLMs) Most PTQ approaches formulate the quantization error based on a calibrated layer-wise $ell$ loss. We propose Output-adaptive (OAC) to incorporate the model output in the calibration process.
  arXiv  Detail & Related papers  (2024-05-23T20:01:17Z)
• Extreme Compression of Large Language Models via Additive Quantization [59.3122859349777]
  Our algorithm, called AQLM, generalizes the classic Additive Quantization (AQ) approach for information retrieval. We provide fast GPU and CPU implementations of AQLM for token generation, which enable us to match or outperform optimized FP16 implementations for speed.
  arXiv  Detail & Related papers  (2024-01-11T18:54:44Z)
• QuIP: 2-Bit Quantization of Large Language Models With Guarantees [44.212441764241]
  This work studies post-training parameter quantization in large language models (LLMs) We introduce quantization with incoherence processing (QuIP), a new method based on the insight that quantization benefits from $textitincoherent$ weight and Hessian matrices.
  arXiv  Detail & Related papers  (2023-07-25T07:44:06Z)
• SqueezeLLM: Dense-and-Sparse Quantization [80.32162537942138]
  Main bottleneck for generative inference with LLMs is memory bandwidth, rather than compute, for single batch inference. We introduce SqueezeLLM, a post-training quantization framework that enables lossless compression to ultra-low precisions of up to 3-bit. Our framework incorporates two novel ideas: (i) sensitivity-based non-uniform quantization, which searches for the optimal bit precision assignment based on second-order information; and (ii) the Dense-and-Sparse decomposition that stores outliers and sensitive weight values in an efficient sparse format.
  arXiv  Detail & Related papers  (2023-06-13T08:57:54Z)
• Improving Convergence for Quantum Variational Classifiers using Weight Re-Mapping [60.086820254217336]
  In recent years, quantum machine learning has seen a substantial increase in the use of variational quantum circuits (VQCs) We introduce weight re-mapping for VQCs, to unambiguously map the weights to an interval of length $2pi$. We demonstrate that weight re-mapping increased test accuracy for the Wine dataset by $10%$ over using unmodified weights.
  arXiv  Detail & Related papers  (2022-12-22T13:23:19Z)
• QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization [54.44028700760694]
  Post-training quantization (PTQ) has driven much attention to produce efficient neural networks without long-time retraining. In this study, we pioneeringly confirm that properly incorporating activation quantization into the PTQ reconstruction benefits the final accuracy. Based on the conclusion, a simple yet effective approach dubbed as QDROP is proposed, which randomly drops the quantization of activations during PTQ.
  arXiv  Detail & Related papers  (2022-03-11T04:01:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.