Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization

• URL: http://arxiv.org/abs/2608.01078v1
• Date: Sun, 02 Aug 2026 08:22:43 GMT
• Title: Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization
• Abstract summary: We propose ScaleQ-1.58, a scalable ternary post-training quantization framework for reasoning LLMs.<n>It is formed by simply integrating AYOT with CAT-Q, which demonstrates several scaling properties.<n>It generalizes well to both dense and MoE architectures, with performance improving as model scale increases.
• Score: 13.903045659324187
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose ScaleQ-1.58, a scalable ternary post-training quantization (PTQ) framework for reasoning LLMs. Its core insight stems from an empirical finding: although modern LLMs are typically trained to exhibit chain-of-thought reasoning capabilities, in the PTQ regime, even the latest CAT-Q method based on learning-based differentiable ternarization still leads to performance collapse on challenging mathematics and coding tasks when using conventional calibration schemes that ignore the model's reasoning process. Driven by this finding, we introduce a simple calibration approach, Attend to Your Own Thoughts (AYOT), where reasoning traces and final answers generated by the pre-trained high-precision target LLM on a proper set of calibration samples are used as the context input during the ternarization process, along with the corresponding questions. ScaleQ-1.58 is formed by simply integrating AYOT with CAT-Q, which demonstrates several scaling properties: (1) with only 4M calibration tokens, Qwen3-1.7B ternarized by ScaleQ-1.58 reaches over 90.52% of the performance of the prior best BitNet b1.58 2B4T averaged over 4 mathematics and coding tasks, and our ternary Qwen3-4B shows an absolute gain of 8.97%, while requiring 1,000,000x fewer calibration tokens for quantization; (2) ScaleQ-1.58 generalizes well to both dense and MoE architectures, with performance improving as model scale increases (up to 235B parameters); (3) ScaleQ-1.58 demonstrates strong generalization across tasks of varying difficulty levels, including mathematics, coding and scientific logic reasoning, as well as commonsense reasoning and basic language generation; (4) its performance continues to improve as the number of calibration tokens increases. Notably, AYOT also exhibits strong generalization ability across other quantization bit-widths. Code will be available at https://github.com/IntelChina-AI/BitTern.

Related papers

• CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs [13.903045659324187]
  We present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs.<n>For pre-trained LLMs with 1.7B to 8B parameters, CAT-Q can efficiently quantize them into ternary models using only 512 calibration samples.<n>We show for the first time that CAT-Q can quantize much larger pre-trained LLMs having 14B to 235B parameters into leading ternary models within just 8 to 60 hours.
  arXiv  Detail & Related papers  (2026-06-25T06:24:02Z)
• AlphaQ: Calibration-Free Bit Allocation for Mixture-of-Experts Quantization [104.10572215799239]
  Mixed-precision quantization can substantially reduce the footprint by assigning different bit-widths to different experts.<n>Existing approaches rely on calibration data to estimate expert importance and determine bit allocation.<n>We propose AlphaQ, a calibration-free bit-allocation method for MoE quantization.
  arXiv  Detail & Related papers  (2026-06-03T15:03:18Z)
• Item Response Scaling Laws: A Measurement Theory Approach for Efficient and Generalizable Neural Scaling Estimation [31.553914838498667]
  We introduce Item Response Scaling Laws (IRSL), a unified framework that integrates Item Response Theory (IRT) within the scaling law framework.<n>IRSL disentangles latent model ability from question characteristics, factorizing the scaling law estimation for $M$ models and $N$ questions.<n>We validate our approach across two prevalent scaling paradigms: (1) pre-training downstream scaling, using 6,612 LM checkpoints and 37,682 questions from 10 benchmarks; and (2) test-time scaling, using 12 LMs and 120 questions from 4 benchmarks with up to 2,500 samples per question.
  arXiv  Detail & Related papers  (2026-05-29T20:55:15Z)
• TernaryLM: Memory-Efficient Language Modeling via Native 1-Bit Quantization with Adaptive Layer-wise Scaling [0.39287497907611874]
  We present TernaryLM, a 132M parameter transformer architecture that employs native 1-bit ternary quantization -1, 0, +1 during training.<n>Our results suggest that native 1-bit training is a promising direction for efficient neural language models.
  arXiv  Detail & Related papers  (2026-02-07T05:35:17Z)
• What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Study [59.44848132298657]
  Post-training quantization (PTQ) usually comes with the cost of large accuracy drops, especially for reasoning tasks under low-bit settings.<n>In this study, we present a systematic empirical study of quantization-aware training (QAT) for reasoning models.
  arXiv  Detail & Related papers  (2026-01-21T11:22:29Z)
• Rethinking Output Alignment For 1-bit Post-Training Quantization of Large Language Models [41.677469535447024]
  Large Language Models (LLMs) deliver strong performance across a wide range of NLP tasks, but their massive sizes hinder deployment on resource-constrained devices.<n>Post-training quantization (PTQ) is widely adopted for its efficiency, as it requires no retraining and only a small dataset for calibration.<n>Recent advances for post-training quantization have demonstrated that even sub-4-bit methods can maintain most of the original model performance.
  arXiv  Detail & Related papers  (2025-12-25T12:39:36Z)
• PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models [64.84734437930362]
  Large Language Models (LLMs) suffer severe performance degradation when facing extremely low-bit (sub 2-bit) quantization.<n>We propose an extremely low-bit PTQ method called PTQ1.61, which enables weight quantization to 1.61-bit for the first time.<n>Experiments indicate our PTQ1.61 achieves state-of-the-art performance in extremely low-bit quantization.
  arXiv  Detail & Related papers  (2025-02-18T08:04:58Z)
• ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantization [73.60493264901359]
  We present a unified framework for rigorous comparisons across 1-bit, 1.58-bit, 2-bit, 3-bit, and 4-bit quantization settings.<n>We show that ternary, 2-bit, and 3-bit quantization maintains comparable performance in the size-accuracy trade-off.<n>Considering hardware constraints, 2-bit quantization offers promising potential for memory reduction and speedup.
  arXiv  Detail & Related papers  (2025-02-04T18:59:26Z)
• EfficientQAT: Efficient Quantization-Aware Training for Large Language Models [50.525259103219256]
  quantization-aware training (QAT) offers a solution by reducing memory consumption through low-bit representations with minimal accuracy loss.<n>We propose Efficient Quantization-Aware Training (EfficientQAT), a more feasible QAT algorithm.<n> EfficientQAT involves two consecutive phases: Block-wise training of all parameters (Block-AP) and end-to-end training of quantization parameters (E2E-QP)
  arXiv  Detail & Related papers  (2024-07-10T17:53:30Z)
• LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit [55.73370804397226]
  Quantization, a key compression technique, can effectively mitigate these demands by compressing and accelerating large language models. We present LLMC, a plug-and-play compression toolkit, to fairly and systematically explore the impact of quantization. Powered by this versatile toolkit, our benchmark covers three key aspects: calibration data, algorithms (three strategies), and data formats.
  arXiv  Detail & Related papers  (2024-05-09T11:49:05Z)
• Zero-Shot Sharpness-Aware Quantization for Pre-trained Language Models [88.80146574509195]
  Quantization is a promising approach for reducing memory overhead and accelerating inference. We propose a novel-aware quantization (ZSAQ) framework for the zero-shot quantization of various PLMs.
  arXiv  Detail & Related papers  (2023-10-20T07:09:56Z)
• Norm Tweaking: High-performance Low-bit Quantization of Large Language Models [21.855106896725598]
  We introduce a technique called norm tweaking, which can be used as a plugin in current PTQ methods to achieve high precision. Our method demonstrates significant improvements in both weight-only quantization and joint quantization of weights and activations. Our simple and effective approach makes it more practical for real-world applications.
  arXiv  Detail & Related papers  (2023-09-06T06:51:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.