GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance

• URL: http://arxiv.org/abs/2505.07004v3
• Date: Sun, 27 Jul 2025 11:06:56 GMT
• Title: GuidedQuant: Large Language Model Quantization via Exploiting End Loss Guidance
• Authors: Jinuk Kim, Marwa El Halabi, Wonpyo Park, Clemens JS Schaefer, Deokjae Lee, Yeonhong Park, Jae W. Lee, Hyun Oh Song,
• Abstract summary: Post-training quantization is a key technique for reducing the memory and inference latency of large language models.<n>We propose GuidedQuant, a novel quantization approach that integrates gradient information from the end loss into the quantization objective.<n> GuidedQuant consistently boosts the performance of state-of-the-art quantization methods across weight-only scalar, weight-only vector, and weight-and-activation quantization.
• Score: 21.134233954419148
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Post-training quantization is a key technique for reducing the memory and inference latency of large language models by quantizing weights and activations without requiring retraining. However, existing methods either (1) fail to account for the varying importance of hidden features to the end loss or, when incorporating end loss, (2) neglect the critical interactions between model weights. To address these limitations, we propose GuidedQuant, a novel quantization approach that integrates gradient information from the end loss into the quantization objective while preserving cross-weight dependencies within output channels. GuidedQuant consistently boosts the performance of state-of-the-art quantization methods across weight-only scalar, weight-only vector, and weight-and-activation quantization. Additionally, we introduce a novel non-uniform scalar quantization algorithm, which is guaranteed to monotonically decrease the quantization objective value, and outperforms existing methods in this category. We release the code at https://github.com/snu-mllab/GuidedQuant.

Related papers

• Low-bit Model Quantization for Deep Neural Networks: A Survey [123.89598730307208]
  This article surveys the recent five-year progress towards low-bit quantization on deep neural networks (DNNs)<n>We discuss and compare the state-of-the-art quantization methods and classify them into 8 main categories and 24 sub-categories according to their core techniques.<n>We shed light on the potential research opportunities in the field of model quantization.
  arXiv  Detail & Related papers  (2025-05-08T13:26:19Z)
• Enhancing Ultra-Low-Bit Quantization of Large Language Models Through Saliency-Aware Partial Retraining [0.0]
  Post-training quantization reduces model size efficiently at the cost of decreased accuracy.<n> quantization-aware training better preserves accuracy but is resource-intensive.<n>We propose an ultra-low-bit quantization method that builds upon ApiQ and extends its performance without the need for full retraining.
  arXiv  Detail & Related papers  (2025-04-14T19:31:21Z)
• Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners [51.32182730502002]
  We introduce Singular-value Diagonal Expansion to refine weight distributions to achieve better quantization alignment.<n>Our plug-and-play weight-quantization methods demonstrate substantial performance improvements over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-07-22T09:45:16Z)
• Compensate Quantization Errors: Make Weights Hierarchical to Compensate Each Other [10.292252814921714]
  We introduce Learnable Singular value Increment (LSI) as an advanced solution to quantization problems. LSI uses Singular Value Decomposition to extract singular values of the weights and make them learnable to help weights compensate each other conditioned on activation. We achieve state-of-the-art performance in diverse quantization settings, no matter in weight-only, weight-activation or extremely low bit scenarios.
  arXiv  Detail & Related papers  (2024-06-24T03:52:52Z)
• BoA: Attention-aware Post-training Quantization without Backpropagation [11.096116957844014]
  Post-training quantization is a promising solution for deploying large language models on resource-constrained devices.<n>We introduce a novel backpropagation-free PTQ algorithm that optimize quantized weights by considering inter-layer dependencies.
  arXiv  Detail & Related papers  (2024-06-19T11:53:21Z)
• WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More [55.0856305773081]
  Large Language Models (LLMs) face significant deployment challenges due to their substantial memory requirements and the computational demands of auto-regressive text generation process. This paper addresses these challenges by focusing on the quantization of LLMs, a technique that reduces memory consumption by converting model parameters and activations into low-bit integers.
  arXiv  Detail & Related papers  (2024-02-19T11:33:21Z)
• AWEQ: Post-Training Quantization with Activation-Weight Equalization for Large Language Models [0.18416014644193066]
  AWEQ excels in both ultra-low-bit quantization and 8-bit weight and activation (W8A8) quantization. We have further refined the equalization method to mitigate quantization bias error, ensuring the robustness of the model.
  arXiv  Detail & Related papers  (2023-11-02T15:18:22Z)
• Attention Round for Post-Training Quantization [0.9558392439655015]
  This paper presents a novel quantification method called Attention Round. The probability of being mapped to different quantified values is negatively correlated with the distance between the quantified values and w, and decay with a Gaussian function. For ResNet18 and MobileNetV2, the post-training quantization proposed in this paper only require 1,024 training data and 10 minutes to complete the quantization process.
  arXiv  Detail & Related papers  (2022-07-07T05:04:21Z)
• Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation [48.838691414561694]
  Nonuniform-to-Uniform Quantization (N2UQ) is a method that can maintain the strong representation ability of nonuniform methods while being hardware-friendly and efficient. N2UQ outperforms state-of-the-art nonuniform quantization methods by 0.71.8% on ImageNet.
  arXiv  Detail & Related papers  (2021-11-29T18:59:55Z)
• In-Hindsight Quantization Range Estimation for Quantized Training [5.65658124285176]
  We propose a simple alternative to dynamic quantization, in-hindsight range estimation, that uses the quantization ranges estimated on previous iterations to quantize the present. Our approach enables fast static quantization of gradients and activations while requiring only minimal hardware support from the neural network accelerator. It is intended as a drop-in replacement for estimating quantization ranges and can be used in conjunction with other advances in quantized training.
  arXiv  Detail & Related papers  (2021-05-10T10:25:28Z)
• Direct Quantization for Training Highly Accurate Low Bit-width Deep Neural Networks [73.29587731448345]
  This paper proposes two novel techniques to train deep convolutional neural networks with low bit-width weights and activations. First, to obtain low bit-width weights, most existing methods obtain the quantized weights by performing quantization on the full-precision network weights. Second, to obtain low bit-width activations, existing works consider all channels equally.
  arXiv  Detail & Related papers  (2020-12-26T15:21:18Z)
• Gradient $\ell_1$ Regularization for Quantization Robustness [70.39776106458858]
  We derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-demand to different bit-widths.
  arXiv  Detail & Related papers  (2020-02-18T12:31:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.