Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin

• URL: http://arxiv.org/abs/2606.09012v1
• Date: Mon, 08 Jun 2026 04:21:39 GMT
• Title: Understanding Quantization-Aware Training: Gradients at Quantized Weights Bias to the Low-Loss Basin
• Abstract summary: Post-training quantization converts a trained full-precision model into low-bit weights without task-level retraining.<n>Although PTQ is efficient and often accurate at moderate bitwidths, it can fail sharply at aggressive bitwidths.<n>We propose a unified geometric framework that explains both PTQ failure and QAT recovery.
• Score: 13.185423069216808
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Post-training quantization (PTQ) converts a trained full-precision model into low-bit weights without task-level retraining, while quantization-aware training (QAT) incorporates quantization into the training loop. Although PTQ is efficient and often accurate at moderate bitwidths, it can fail sharply at aggressive bitwidths; QAT is more expensive but can often recover the lost accuracy. We propose a unified geometric framework that explains both PTQ failure and QAT recovery. We model full-precision training as following a low-loss \emph{river} inside a wider \emph{valley}: a normal neighborhood of the river forms a nearly flat \emph{basin}, while leaving this basin incurs a sharp loss increase. When the quantization grid is comparable to the basin width, local PTQ objectives, including rounding and Hessian-based second-order reconstruction, can select a high-loss deployed quantized point outside the basin even when nearby low-loss quantized points exist. In this regime, straight-through-estimator-based QAT has a useful bias: it evaluates gradients at the deployed quantized weights while updating latent full-precision weights, causing the gradient to sense the valley wall and acquire an inward component that steers subsequent quantized iterates back into the basin. We formalize this mechanism through a local landscape model, construct a geometric PTQ failure mode, and prove finite-time QAT recovery under local quantizer-compatibility assumptions. Experiments across vision and language models under multiple neural-network quantization schemes corroborate the predicted basin-crossing failure of PTQ and the corresponding recovery mechanism of QAT.

Related papers

• QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction [29.200349824129102]
  QUASAR continuously performs loss-aware reconstruction in the training loop to lower the loss floor and improve the resulting low-bit model.<n>It achieves the lowest held-out KL divergence among competitive QAT methods at 2, 3, and 4 bits.<n>It improves average accuracy across eight tasks by 3.5-4.3 percentage points over strong QAT and PTQ baselines.
  arXiv  Detail & Related papers  (2026-08-14T05:29:58Z)
• ReQuant: Fixed-Grid Discrete Refinement for Post-Training Quantization [10.647335642551193]
  Post-training quantization (PTQ) is widely used to reduce the computational cost of large language models.<n>We introduce ReQuant, a backpropagation-free fixed-grid refinement procedure for this stage.<n>We show that ReQuant consistently improves quantized models from heterogeneous PTQ initializers.
  arXiv  Detail & Related papers  (2026-08-07T09:27:14Z)
• Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization [12.719434226050522]
  Low-bit quantization suffers severe accuracy degradation on compact networks.<n> Conventional pipelines adopt monolithic optimization.<n>We propose Normalization Affine Preconditioning (NAP) for targeted subspace optimization.
  arXiv  Detail & Related papers  (2026-08-04T16:50:04Z)
• Efficient Tuning Before Low-Bit Post-Training Quantization for Stochastic Gradient Descent-optimized Models [8.779009700879982]
  Post-training quantization (PTQ) compresses deep neural networks for deployment under limited memory and computational budgets.<n>Most existing PTQ methods operate on an unconstrained full-precision (FP) model and primarily address quantization errors through post-hoc reconstruction.<n>We argue that low-bit PTQ accuracy is limited not only by post-quantization error minimization, but also by the quantization-error tolerance of FP model itself.
  arXiv  Detail & Related papers  (2026-07-13T10:24:36Z)
• LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs [52.1276403258812]
  We introduce Logit-aware Final-block Quantization (LFQ), a simple yet effective enhancement to block-wise PTQ.<n>LFQ consistently improves the accuracy of complex generation tasks over state-of-the-art block-wise PTQ.
  arXiv  Detail & Related papers  (2026-05-28T11:02:23Z)
• Saliency-Aware Regularized Quantization Calibration for Large Language Models [12.382772717031772]
  Post-training quantization (PTQ) is an effective approach for deploying large language models (LLMs) under memory and latency constraints.<n>Most existing PTQ methods determine quantization parameters by minimizing a layer-wise reconstruction error on a predetermined calibration dataset.<n>The proposed regularization encourages quantized weights to remain close to the original weights during calibration, leading to improved generalization at inference time.
  arXiv  Detail & Related papers  (2026-05-07T05:33:26Z)
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost [53.25965863436039]
  Quantization-aware training (QAT) provides a more principled solution, but its reliance on backpropagation incurs prohibitive memory costs.<n>We propose ZeroQAT, a zeroth-order optimization-based QAT framework that supports both weight and activation quantization.<n>Experiments show that ZeroQAT consistently outperforms representative PTQ and QAT baselines while requiring significantly less memory.
  arXiv  Detail & Related papers  (2025-08-21T01:18:27Z)
• FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation [55.12070409045766]
  Post-training quantization (PTQ) has stood out as a cost-effective and promising model compression paradigm in recent years.<n>Current PTQ methods for Vision Transformers (ViTs) still suffer from significant accuracy degradation, especially under low-bit quantization.
  arXiv  Detail & Related papers  (2025-06-13T07:57:38Z)
• Quantization Error Propagation: Revisiting Layer-Wise Post-Training Quantization [0.0]
  Layer-wise PTQ is a promising technique for compressing large language models (LLMs)<n>Recent progress in this area is saturating, underscoring the need to revisit its core limitations and explore further improvements.<n>We propose Quantization Error Propagation (QEP), a general, lightweight, and scalable framework that enhances layer-wise PTQ by explicitly propagating quantization errors and compensating for accumulated errors.
  arXiv  Detail & Related papers  (2025-04-13T15:56:00Z)
• APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers [71.2294205496784]
  We propose textbfAPHQ-ViT, a novel PTQ approach based on importance estimation with Average Perturbation Hessian (APH)<n>We show that APHQ-ViT using linear quantizers outperforms existing PTQ methods by substantial margins in 3-bit and 4-bit across different vision tasks.
  arXiv  Detail & Related papers  (2025-04-03T11:48:56Z)
• PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models [64.84734437930362]
  Large Language Models (LLMs) suffer severe performance degradation when facing extremely low-bit (sub 2-bit) quantization.<n>We propose an extremely low-bit PTQ method called PTQ1.61, which enables weight quantization to 1.61-bit for the first time.<n>Experiments indicate our PTQ1.61 achieves state-of-the-art performance in extremely low-bit quantization.
  arXiv  Detail & Related papers  (2025-02-18T08:04:58Z)
• OAC: Output-adaptive Calibration for Accurate Post-training Quantization [28.67781845829386]
  Post-training Quantization (PTQ) techniques have been developed to compress Large Language Models (LLMs)<n>Most PTQ approaches formulate the quantization error based on a layer-wise Euclidean loss, ignoring the model output.<n>We propose Output-adaptive Quantization (OAC) to incorporate the model output in the calibration process.
  arXiv  Detail & Related papers  (2024-05-23T20:01:17Z)
• QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization [54.44028700760694]
  Post-training quantization (PTQ) has driven much attention to produce efficient neural networks without long-time retraining. In this study, we pioneeringly confirm that properly incorporating activation quantization into the PTQ reconstruction benefits the final accuracy. Based on the conclusion, a simple yet effective approach dubbed as QDROP is proposed, which randomly drops the quantization of activations during PTQ.
  arXiv  Detail & Related papers  (2022-03-11T04:01:53Z)
• A Statistical Framework for Low-bitwidth Training of Deep Neural Networks [70.77754244060384]
  Fully quantized training (FQT) uses low-bitwidth hardware by quantizing the activations, weights, and gradients of a neural network model. One major challenge with FQT is the lack of theoretical understanding, in particular of how gradient quantization impacts convergence properties.
  arXiv  Detail & Related papers  (2020-10-27T13:57:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.