Why Does Post-Training Quantization Work?

• URL: http://arxiv.org/abs/2609.11716v1
• Date: Thu, 10 Sep 2026 15:32:32 GMT
• Title: Why Does Post-Training Quantization Work?
• Abstract summary: Post-training quantization compresses large language models by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states.<n>This raises the question: why does post-training quantization work?<n>We identify two mechanisms that characterize pretrained quantization robustness.
• Score: 24.890051501757146
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Post-training quantization compresses large language models (LLMs) by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states. Naively, these errors should accumulate with depth and corrupt next-token prediction; randomly initialized models accumulate these discrepancies rapidly, whereas quantized pretrained models accumulate much less hidden-state error and largely maintain downstream task performance, even though they were never trained with quantization noise. This raises the question we address: why does post-training quantization work? Comparing full-precision and quantized forward passes, we identify two mechanisms that characterize pretrained quantization robustness. First, the error a layer newly introduces tends to oppose the error it inherits from the layer's input. The two cancel partially such that the discrepancy between full-precision and quantized passes grows slowly. This counteracting residual interaction develops during pretraining. Our quantitative analysis identifies it as a major factor slowing hidden-error growth. Second, LM-head geometry preferentially preserves the scores and probabilities of high-ranked tokens, which typically represent the model's most confident predictions. Together, these mechanisms explain why quantization error that passes through numerous layers can still produce only small output changes, and we verify the findings across models and quantization settings.

Related papers

• Quantization Degradation in Large Language Models: A Signal-Noise Perspective [31.171218460791483]
  We systematically study weight-only post-training quantization across bit-widths, quantization methods, model scales and downstream tasks.<n>We observe that such degradation varies substantially across these factors.<n>We use the signal-to-noise ratio (SNR) to measure how strongly quantization perturbs full-precision representations.
  arXiv  Detail & Related papers  (2026-08-08T15:28:45Z)
• Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models [19.14957817129379]
  We show that low-bit post-training quantization can introduce a hidden test-time compute cost.<n>We find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup.
  arXiv  Detail & Related papers  (2026-06-24T07:54:55Z)
• Enhancing Post-Training Quantization via Future Activation Awareness [84.76726857601753]
  Post-training quantization (PTQ) is a widely used method to compress large language models (LLMs) without fine-tuning.<n>We propose Future-Aware Quantization (FAQ), which leverages future-layer activations to guide quantization.<n>FAQ consistently outperforms prior methods with negligible extra cost, requiring no backward passes, data reconstruction, or tuning.
  arXiv  Detail & Related papers  (2026-01-28T12:03:30Z)
• Why Do Some Inputs Break Low-Bit LLM Quantization? [33.31258645346889]
  Low-bit weight-only quantization significantly reduces the memory footprint of large language models (LLMs)<n>We analyze diverse 3-4 bit methods on LLMs ranging from 7B-70B in size and find that the quantization errors of 50 pairs of methods are strongly correlated (avg. 0.82) on FineWeb examples.
  arXiv  Detail & Related papers  (2025-05-24T16:17:50Z)
• Low-bit Model Quantization for Deep Neural Networks: A Survey [123.89598730307208]
  This article surveys the recent five-year progress towards low-bit quantization on deep neural networks (DNNs)<n>We discuss and compare the state-of-the-art quantization methods and classify them into 8 main categories and 24 sub-categories according to their core techniques.<n>We shed light on the potential research opportunities in the field of model quantization.
  arXiv  Detail & Related papers  (2025-05-08T13:26:19Z)
• Compute-Optimal LLMs Provably Generalize Better With Scale [102.29926217670926]
  We develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime.<n>We introduce a novel, fully empirical Freedman-type martingale concentration that tightens existing bounds by accounting for the variance of the loss function.<n>We produce a scaling law for the generalization gap, with bounds that become predictably stronger with scale.
  arXiv  Detail & Related papers  (2025-04-21T16:26:56Z)
• Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners [51.32182730502002]
  We introduce Singular-value Diagonal Expansion to refine weight distributions to achieve better quantization alignment.<n>Our plug-and-play weight-quantization methods demonstrate substantial performance improvements over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-07-22T09:45:16Z)
• Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction [48.740630807085566]
  Post-training quantization (PTQ) for vision transformers (ViTs) has received increasing attention from both academic and industrial communities.<n>Current methods fail to account for the complex interactions between quantized weights and activations, resulting in significant quantization errors and suboptimal performance.<n>This paper presents ERQ, an innovative two-step PTQ method specifically crafted to reduce quantization errors arising from activation and weight quantization sequentially.
  arXiv  Detail & Related papers  (2024-07-09T12:06:03Z)
• Regularized Vector Quantization for Tokenized Image Synthesis [126.96880843754066]
  Quantizing images into discrete representations has been a fundamental problem in unified generative modeling. deterministic quantization suffers from severe codebook collapse and misalignment with inference stage while quantization suffers from low codebook utilization and reconstruction objective. This paper presents a regularized vector quantization framework that allows to mitigate perturbed above issues effectively by applying regularization from two perspectives.
  arXiv  Detail & Related papers  (2023-03-11T15:20:54Z)
• Qu-ANTI-zation: Exploiting Quantization Artifacts for Achieving Adversarial Outcomes [5.865029600972316]
  Quantization is a technique that transforms the parameter representation of a neural network from floating-point numbers into lower-precision ones. We propose a new training framework to implement adversarial quantization outcomes. We show that a single compromised model defeats multiple quantization schemes.
  arXiv  Detail & Related papers  (2021-10-26T10:09:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.