KronQ: LLM Quantization via Kronecker-Factored Hessian

• URL: http://arxiv.org/abs/2607.07964v1
• Date: Wed, 08 Jul 2026 22:34:52 GMT
• Title: KronQ: LLM Quantization via Kronecker-Factored Hessian
• Abstract summary: Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining.<n>We propose KronQ, a PTQ framework that challenges the assumption by introducing the gradient covariance into the quantization pipeline.
• Score: 19.710403142085102
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Post-training quantization (PTQ) is a widely adopted technique for compressing large language models (LLMs) without retraining. Existing second-order PTQ methods, including GPTQ, construct quantization objectives exclusively from input activation statistics, effectively assuming that all output channels contribute equally to the layer-wise reconstruction objective. We propose KronQ, a PTQ framework that challenges this assumption by introducing the gradient covariance into the quantization pipeline. Under the Kronecker-factored Hessian approximation, the quantization loss depends jointly on both the activation and gradient covariances, and KronQ exploits this at two complementary levels. (1) KronQ introduces bidirectional incoherence processing, extending the existing input-side random rotation to the output dimension using the gradient covariance, reducing weight magnitude variance across both input and output dimensions. (2) KronQ derives a new sensitivity metric for inter-layer mixed-precision allocation, driven by the gradient and activation Hessian traces. Notably, in the case of 2-bit weight-only quantization on LLaMA-3-70B, while GPTQ and GPTAQ diverge or produce degenerate quantizations (>2000 perplexity on WikiText-2), KronQ achieves 7.93 perplexity.

Related papers

• UniSVQ: 2-bit Unified Scalar-Vector Quantization [48.190936990215384]
  Post-training quantization at the 2-bit level enables low-cost deployment and inference acceleration for large language models.<n>We propose UniSVQ, a unified 2-bit quantization framework that bridges scalar and vector quantization by parameterizing codewords as an affine transform of integer lattices.
  arXiv  Detail & Related papers  (2026-06-09T07:50:52Z)
• LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs [52.1276403258812]
  We introduce Logit-aware Final-block Quantization (LFQ), a simple yet effective enhancement to block-wise PTQ.<n>LFQ consistently improves the accuracy of complex generation tasks over state-of-the-art block-wise PTQ.
  arXiv  Detail & Related papers  (2026-05-28T11:02:23Z)
• GSQ: Highly-Accurate Low-Precision Scalar Quantization for LLMs via Gumbel-Softmax Sampling [36.47926569464477]
  We introduce GSQ (Gumbel-Softmax Quantization), a post-training scalar quantization method which jointly learns the per-coordinate grid assignments and the per-group scales.<n>On the standard Llama-3.1-8B/70B-Instruct models, GSQ closes most of the gap between scalar quantization and the QTIP frontier at 2 and 3 bits.<n>We show that GSQ scales to trillion-scale Mixture-of-Experts models such as Kimi-K2.5, where vector-quantized methods are difficult to apply.
  arXiv  Detail & Related papers  (2026-04-20T17:45:47Z)
• Rethinking Practical and Efficient Quantization Calibration for Vision-Language Models [11.411411301593011]
  Post-training quantization (PTQ) is a primary approach for deploying large language models without fine-tuning.<n>We propose the Token-level Importance-aware Layer-wise Quantization framework (TLQ)<n> TLQ is evaluated across two models, three model scales, and two quantization settings, consistently achieving performance improvements across all settings.
  arXiv  Detail & Related papers  (2026-02-08T10:19:25Z)
• BPDQ: Bit-Plane Decomposition Quantization on a Variable Grid for Large Language Models [56.504879072674015]
  We propose Bit-Plane Decomposition Quantization (BPDQ), which constructs a variable quantization grid via bit-planes and scalar coefficients.<n>BPDQ enables serving Qwen2.5-72B on a single GTX 3090 with 83.85% GSM8K accuracy (vs. 90.83% at 16-bit)
  arXiv  Detail & Related papers  (2026-02-04T02:54:37Z)
• Cat: Post-Training Quantization Error Reduction via Cluster-based Affine Transformation [47.791962198275066]
  Post-Training Quantization (PTQ) reduces the memory footprint and computational overhead of deep neural networks by converting full-precision (FP) values into quantized and compressed data types.<n>While PTQ is more cost-efficient than Quantization-Aware Training (QAT), it is highly susceptible to accuracy degradation under a low-bit quantization regime.<n>We propose Cluster-based Affine Transformation (CAT), an error-reduction framework that employs cluster-specific parameters to align LQ outputs with FP counterparts.
  arXiv  Detail & Related papers  (2025-09-30T14:00:28Z)
• SINQ: Sinkhorn-Normalized Quantization for Calibration-Free Low-Precision LLM Weights [8.95245917088986]
  Post-training quantization has emerged as the most widely used strategy for deploying large language models at low precision.<n>Current methods show perplexity degradation at bit-widths less than or equal to 4, partly because representing outliers causes precision issues.<n>We introduce SINQ to augment existing post-training quantizers with an additional second-axis scale factor and a fast Sinkhorn-Knopp-style algorithm.
  arXiv  Detail & Related papers  (2025-09-26T21:22:54Z)
• LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generation [41.66473889057111]
  Diffusion Transformers (DiTs) have achieved impressive performance in text-to-image and text-to-video generation.<n>DiTs' high computational cost and large parameter sizes pose significant challenges for usage in resource-constrained scenarios.<n>We propose LRQ-DiT, an efficient and accurate post-training quantization framework for image and video generation.
  arXiv  Detail & Related papers  (2025-08-05T14:16:11Z)
• PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models [64.84734437930362]
  Large Language Models (LLMs) suffer severe performance degradation when facing extremely low-bit (sub 2-bit) quantization.<n>We propose an extremely low-bit PTQ method called PTQ1.61, which enables weight quantization to 1.61-bit for the first time.<n>Experiments indicate our PTQ1.61 achieves state-of-the-art performance in extremely low-bit quantization.
  arXiv  Detail & Related papers  (2025-02-18T08:04:58Z)
• Extending Quantum Perceptrons: Rydberg Devices, Multi-Class Classification, and Error Tolerance [67.77677387243135]
  Quantum Neuromorphic Computing (QNC) merges quantum computation with neural computation to create scalable, noise-resilient algorithms for quantum machine learning (QML) At the core of QNC is the quantum perceptron (QP), which leverages the analog dynamics of interacting qubits to enable universal quantum computation.
  arXiv  Detail & Related papers  (2024-11-13T23:56:20Z)
• Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction [48.740630807085566]
  Post-training quantization (PTQ) for vision transformers (ViTs) has received increasing attention from both academic and industrial communities.<n>Current methods fail to account for the complex interactions between quantized weights and activations, resulting in significant quantization errors and suboptimal performance.<n>This paper presents ERQ, an innovative two-step PTQ method specifically crafted to reduce quantization errors arising from activation and weight quantization sequentially.
  arXiv  Detail & Related papers  (2024-07-09T12:06:03Z)
• OAC: Output-adaptive Calibration for Accurate Post-training Quantization [28.67781845829386]
  Post-training Quantization (PTQ) techniques have been developed to compress Large Language Models (LLMs)<n>Most PTQ approaches formulate the quantization error based on a layer-wise Euclidean loss, ignoring the model output.<n>We propose Output-adaptive Quantization (OAC) to incorporate the model output in the calibration process.
  arXiv  Detail & Related papers  (2024-05-23T20:01:17Z)
• CBQ: Cross-Block Quantization for Large Language Models [66.82132832702895]
  Post-training quantization (PTQ) has played a key role in compressing large language models (LLMs) with ultra-low costs.<n>We propose CBQ, a cross-block reconstruction-based PTQ method for LLMs.<n> CBQ employs a cross-block dependency using a reconstruction scheme, establishing long-range dependencies across multiple blocks to minimize error accumulation.
  arXiv  Detail & Related papers  (2023-12-13T07:56:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.