Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models

• URL: http://arxiv.org/abs/2606.25519v2
• Date: Mon, 29 Jun 2026 20:25:31 GMT
• Title: Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models
• Abstract summary: We show that low-bit post-training quantization can introduce a hidden test-time compute cost.<n>We find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup.
• Score: 19.14957817129379
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quantization is widely used to reduce the inference cost of large language models, but its effect on reasoning models is not fully captured by final-answer accuracy or per-token latency. We show that low-bit post-training quantization can introduce a hidden test-time compute cost: quantized reasoning models often generate longer chains of thought even when they still answer correctly. Across mathematical reasoning, code generation, scientific question answering, and agentic tool-use benchmarks, we find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup. To measure this effect, we introduce the CoT Token Inflation Ratio, which compares reasoning length between quantized and full-precision models averaged across all evaluation benchmarks. We further show that token inflation is accompanied by behavioral changes in the reasoning trace, including more intermediate steps and greater semantic repetition. These changes translate into measurable end-to-end real-world serving penalties. Finally, we evaluate mitigation strategies and find that prompting and decoding-time sampling offer inconsistent accuracy-length trade-offs, while quantization-aware training shows more promise in reducing both accuracy degradation and token inflation. Our results suggest that reasoning-token usage should be reported alongside accuracy when evaluating quantized reasoning models.

Related papers

• Why Does Post-Training Quantization Work? [24.890051501757146]
  Post-training quantization compresses large language models by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states.<n>This raises the question: why does post-training quantization work?<n>We identify two mechanisms that characterize pretrained quantization robustness.
  arXiv  Detail & Related papers  (2026-09-10T15:32:32Z)
• Extreme Low-Bit Inference in Reasoning Models: Failure Modes and Targeted Recovery [40.94400211806987]
  We show that aggressive 2-bit inference can fail to deliver end-to-end speedup because instability in the generation process inflates total token count.<n>We analyze full reasoning traces of Qwen3 reasoning models across mathematical and commonsense benchmarks.<n>Our results show that extreme low-bit reasoning becomes practical when its failures are treated as controllable generation pathologies.
  arXiv  Detail & Related papers  (2026-06-01T10:04:09Z)
• Quantized Reasoning Models Think They Need to Think Longer, but They Do Not [20.88617888340002]
  Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood.<n>Across math, coding, and science QA, we find that aggressive PTQ reduces accuracy while increasing chain-of-thought (CoT) length.<n>In up to 52% of the quantized models' failures, models reach the right answer in intermediate reasoning steps but do not output it as a final answer.
  arXiv  Detail & Related papers  (2026-05-29T18:00:00Z)
• Large Language Models Decide Early and Explain Later [47.20810161393936]
  We study the evolution of predicted answers over reasoning steps using forced answer completion.<n>We show that simples, including probe-based stopping, can reduce reasoning token usage by 500 tokens per query.
  arXiv  Detail & Related papers  (2026-04-24T06:26:24Z)
• Decomposing Reasoning Efficiency in Large Language Models [2.4149105714758545]
  We decompose token efficiency into interpretable factors: completion under a fixed token budget, conditional correctness given completion, and verbosity.<n>When reasoning traces are available, we add deterministic trace-quality measures to separate looping from verbose-but-engaged reasoning.<n>Our decomposition reveals distinct bottleneck profiles that suggest different efficiency interventions.
  arXiv  Detail & Related papers  (2026-02-10T14:09:18Z)
• Arbitrage: Efficient Reasoning via Advantage-Aware Speculation [71.45710345765528]
  Speculative Decoding accelerates inference by employing a fast but inaccurate draft model to autoregressively propose tokens.<n>But due to unnecessary rejections caused by token mismatches in semantically equivalent steps, traditional token-level Speculative Decoding struggles in reasoning tasks.<n>We propose Arbitrage, a novel step-level speculative generation framework that routes generation dynamically based on the relative advantage between draft and target models.
  arXiv  Detail & Related papers  (2025-12-04T17:50:53Z)
• TokenSqueeze: Performance-Preserving Compression for Reasoning LLMs [57.217593337454026]
  TokenSqueeze is a novel Long2Short method that condenses reasoning paths while preserving performance and relying exclusively on self-generated data.<n>We show that TokenSqueeze reduces token usage while maintaining accuracy on the MATH500 benchmark.
  arXiv  Detail & Related papers  (2025-11-17T10:38:56Z)
• Learning to Reason Efficiently with Discounted Reinforcement Learning [57.28725125116862]
  Large reasoning models (LRMs) often consume excessive tokens, inflating computational cost and latency.<n>We challenge the assumption that longer responses improve accuracy.<n>By penalizing reasoning tokens using a discounted reinforcement learning setup (interpretable as a small token cost) and analyzing Blackwell optimality in restricted policy classes, we encourage concise yet accurate reasoning.
  arXiv  Detail & Related papers  (2025-10-27T16:17:45Z)
• Entropy After $\langle \texttt{/Think} \rangle$ for reasoning model early exiting [38.93424884988798]
  We show that large reasoning models overthink, continuing to revise answers even after reaching the correct solution.<n>We propose Entropy After /Think> (EAT) for monitoring and deciding whether to exit reasoning early.<n>EAT reduces token usage by 13 - 21% without harming accuracy.
  arXiv  Detail & Related papers  (2025-09-30T16:59:37Z)
• Broken Tokens? Your Language Model can Secretly Handle Non-Canonical Tokenizations [83.93566096400723]
  We find that instruction-tuned models retain up to 93.4% of their original performance when given a randomly sampled tokenization.<n>Character-level segmentation improves string manipulation and code understanding tasks by up to +14%.<n>Right-aligned digit grouping enhances large-number arithmetic by +33%.
  arXiv  Detail & Related papers  (2025-06-23T18:02:26Z)
• Compute-Optimal LLMs Provably Generalize Better With Scale [102.29926217670926]
  We develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime.<n>We introduce a novel, fully empirical Freedman-type martingale concentration that tightens existing bounds by accounting for the variance of the loss function.<n>We produce a scaling law for the generalization gap, with bounds that become predictably stronger with scale.
  arXiv  Detail & Related papers  (2025-04-21T16:26:56Z)
• Token-Supervised Value Models for Enhancing Mathematical Problem-Solving Capabilities of Large Language Models [56.32800938317095]
  Existing verifiers are sub-optimal for tree search techniques at test time.<n>We propose token-supervised value models (TVMs)<n>TVMs assign each token a probability that reflects the likelihood of reaching the correct final answer.
  arXiv  Detail & Related papers  (2024-07-12T13:16:50Z)
• AQD: Towards Accurate Fully-Quantized Object Detection [94.06347866374927]
  We propose an Accurate Quantized object Detection solution, termed AQD, to get rid of floating-point computation. Our AQD achieves comparable or even better performance compared with the full-precision counterpart under extremely low-bit schemes.
  arXiv  Detail & Related papers  (2020-07-14T09:07:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.