Quantized Reasoning Models Think They Need to Think Longer, but They Do Not

• URL: http://arxiv.org/abs/2606.00206v1
• Date: Fri, 29 May 2026 18:00:00 GMT
• Title: Quantized Reasoning Models Think They Need to Think Longer, but They Do Not
• Abstract summary: Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood.<n>Across math, coding, and science QA, we find that aggressive PTQ reduces accuracy while increasing chain-of-thought (CoT) length.<n>In up to 52% of the quantized models' failures, models reach the right answer in intermediate reasoning steps but do not output it as a final answer.
• Score: 20.88617888340002
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Post-training quantization (PTQ) is widely used to deploy large language models efficiently, but its effect on reasoning models is not well understood. Across math, coding, and science QA, we find that aggressive PTQ reduces accuracy while increasing chain-of-thought (CoT) length. Surprisingly, we show that in up to 52% of the quantized models' failures, models reach the right answer in intermediate reasoning steps but do not output it as a final answer. To understand why quantization leads to this increase in overthinking errors, we measure the token-level KL divergence between quantized and full-precision output distributions. Positions with high KL divergence correlate strongly with high next-token entropy, and at these positions quantized models disproportionately sample overthinking markers such as "wait", "but", and "alternatively". We show that simply introducing a training-free logit penalty on a curated set of overthinking markers can reduce CoT length by 12--23% while preserving or improving accuracy across 5 models (1.5B-32B parameters), 3 quantization methods, and 5 benchmarks, yielding a favorable Pareto frontier of accuracy against reasoning cost compared to penalizing other token sets. Overthinking errors produced by quantized models are particularly reduced by up to 58%.

Related papers

• Why Does Post-Training Quantization Work? [24.890051501757146]
  Post-training quantization compresses large language models by storing their weights at reduced precision, and each quantized weight introduces an error into the hidden states.<n>This raises the question: why does post-training quantization work?<n>We identify two mechanisms that characterize pretrained quantization robustness.
  arXiv  Detail & Related papers  (2026-09-10T15:32:32Z)
• Quantization Inflates Reasoning: Token Inflation as a Hidden Cost of Low-Bit Reasoning Models [19.14957817129379]
  We show that low-bit post-training quantization can introduce a hidden test-time compute cost.<n>We find that INT4/INT3 quantization can preserve accuracy but increase reasoning-token usage, offsetting the expected per-token speedup.
  arXiv  Detail & Related papers  (2026-06-24T07:54:55Z)
• LFQ: Logit-aware Final-block Quantization for Boosting the Generation Quality of Low-Bit Quantized LLMs [52.1276403258812]
  We introduce Logit-aware Final-block Quantization (LFQ), a simple yet effective enhancement to block-wise PTQ.<n>LFQ consistently improves the accuracy of complex generation tasks over state-of-the-art block-wise PTQ.
  arXiv  Detail & Related papers  (2026-05-28T11:02:23Z)
• Understanding and Mitigating Premature Confidence for Better LLM Reasoning [76.16007941549857]
  Long chains of thought (CoT) from current language models frequently contain logical gaps and unjustified leaps.<n>We find such a signal in how the model's confidence evolves during reasoning.<n>We exploit this in progressive confidence shaping, a reinforcement learning objective that trains models to update their confidence as they reason rather than commit early.
  arXiv  Detail & Related papers  (2026-05-23T04:42:45Z)
• Thinking in Latents: Adaptive Anchor Refinement for Implicit Reasoning in LLMs [3.0215547730060206]
  We introduce AdaAnchor, a latent reasoning framework that performs silent iterative computation by refining a set of latent anchor vectors attached to the input.<n>AdaAnchor with adaptive halting yields accuracy gains of up to 5% over fixed-step latent refinement while reducing average latent refinement steps by 48-60% under the same maximum-step budget.
  arXiv  Detail & Related papers  (2026-03-16T10:06:05Z)
• Stepwise Penalization for Length-Efficient Chain-of-Thought Reasoning [66.22060690012512]
  Large reasoning models improve with more test-time computation, but often overthink, producing unnecessarily long chains-of-thought that raise cost without improving accuracy.<n>We propose Step-wise Adaptive Penalization (SWAP), a fine-grained framework that allocates length reduction across steps based on intrinsic contribution.
  arXiv  Detail & Related papers  (2026-02-27T20:23:59Z)
• Think Deep, Not Just Long: Measuring LLM Reasoning Effort via Deep-Thinking Tokens [12.788799173865]
  We quantify inference-time effort by identifying deep-thinking tokens.<n>Think@n is a test-time scaling strategy that prioritizes samples with high deep-thinking ratios.
  arXiv  Detail & Related papers  (2026-02-13T23:07:37Z)
• Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning [11.179446105672461]
  We propose a multi-stage efficient reasoning method that combines supervised fine-tuning and reinforcement learning.<n>Our approach reduces response length by an average of 28% for 8B models and 40% for 32B models.<n>It achieves a superior trade-off compared to more complex state-of-the-art efficient reasoning methods.
  arXiv  Detail & Related papers  (2026-01-06T12:31:51Z)
• Your Models Have Thought Enough: Training Large Reasoning Models to Stop Overthinking [50.97239453902612]
  Large Reasoning Models (LRMs) have achieved impressive performance on challenging tasks, yet their deep reasoning often incurs substantial computational costs.<n>Inspired by Evidence Accumulation Models, we find that LRMs have accumulated sufficient information early in reasoning, making further reasoning steps redundant.<n>We propose Just-Enough Thinking (JET), which trains models to proactively terminate unnecessary reasoning.
  arXiv  Detail & Related papers  (2025-09-27T16:25:06Z)
• What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decoding [84.42056293290015]
  We analyze the token-level misalignment between reasoning and non-reasoning models.<n>Motivated by the Local Misalignment Diminish, we propose FoReaL-Decoding.<n>On four popular math-reasoning benchmarks, FoReaL-Decoding reduces theoretical FLOPs by 30 to 50% and trims CoT length by up to 40%.
  arXiv  Detail & Related papers  (2025-06-08T05:08:32Z)
• The Price of a Second Thought: On the Evaluation of Reasoning Efficiency in Large Language Models [54.88805865447848]
  We show that instruct models achieve higher efficiency overall, and problem difficulty affects efficiency.<n>We propose COTHINK, a simple two-stage pipeline: an instruct model drafts a brief outline, and a thinking model expands it.<n>On GSM8K, MATH500, and AIME24, COTHINK cuts token usage by 21.1% while keeping accuracy on four thinking models, and remains competitive with strong efficiency baselines.
  arXiv  Detail & Related papers  (2025-05-28T06:24:45Z)
• Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning [39.56908863102256]
  Low-bit post-training quantization impairs mathematical reasoning up to 69.81% in harder settings.<n>We address two deployment-critical questions with process-level precision.<n>In our settings, as few as 332 curated examples and 3--5 minutes of compute on a single GPU recover 4-bit weight math reasoning toward the full-precision baseline.
  arXiv  Detail & Related papers  (2025-05-16T12:11:40Z)
• Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models [48.98109982725689]
  We conduct the first systematic study on quantized reasoning models.<n>Our investigation covers weight, KV cache, and activation quantization using state-of-the-art algorithms at varying bit-widths.<n>We identify model size, model origin, and task difficulty as critical determinants of performance.
  arXiv  Detail & Related papers  (2025-04-07T08:22:45Z)
• Scaling Laws For Mixed Quantization [14.27345780977423]
  Post-training quantization of Large Language Models (LLMs) has proven effective in reducing memory and computational requirements for inference.<n>We introduce two critical metrics, named the quantization ratio ($Q_r$) and quantization block size ($Q_b$)<n>We propose a unified scaling law on post-training quantization (PTQ) that can predict loss degeneration for varying $Q_r$ and $Q_b$.
  arXiv  Detail & Related papers  (2024-10-09T09:45:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.