Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning

• URL: http://arxiv.org/abs/2501.03035v4
• Date: Mon, 24 Feb 2025 14:34:37 GMT
• Title: Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoning
• Authors: Zhen Li, Yupeng Su, Runming Yang, Congkai Xie, Zheng Wang, Zhongwei Xie, Ngai Wong, Hongxia Yang,
• Abstract summary: Large language models have achieved significant advancements in complex mathematical reasoning benchmarks, such as MATH.<n>Model quantization has emerged as an effective strategy to reduce memory usage and computational costs by employing lower precision and bit-width representations.
• Score: 29.687113675756127
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models have achieved significant advancements in complex mathematical reasoning benchmarks, such as MATH. However, their substantial computational requirements present challenges for practical deployment. Model quantization has emerged as an effective strategy to reduce memory usage and computational costs by employing lower precision and bit-width representations. In this study, we systematically evaluate the impact of quantization on mathematical reasoning tasks. Our results demonstrate that aggressive quantization methods like AWQ and GPTQ introduce up to 32.39% accuracy degradation (average 11.31%) on Llama-3 models, particularly in numerical computation and reasoning planning. To address this, we introduce a multidimensional evaluation framework combining qualitative capability analysis and quantitative error assessment. We further develop targeted recovery strategies, showing that fine-tuning quantized models on only 545 task-specific examples for 3 minutes on 4 GPUs effectively restores reasoning capabilities to near full-precision levels. Additionally, our error assessment pipeline achieves 98.9% accuracy in diagnosing and localizing errors across 3,366 failure cases, providing actionable insights for mitigating quantization-induced degradation.

Related papers

• Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models [48.98109982725689]
  We conduct the first systematic study on quantized reasoning models, evaluating the open-sourced DeepSeek-R1-Distilled Qwen and LLaMA families. Our investigation covers weight, KV cache, and activation quantization using state-of-the-art algorithms at varying bit-widths. We identify model size, model origin, and task difficulty as critical determinants of performance.
  arXiv  Detail & Related papers  (2025-04-07T08:22:45Z)
• Improving Quantization with Post-Training Model Expansion [0.35377121774178694]
  Post-training model expansion is a viable strategy to improve model quality within a quantization co-design space. We show it is possible to progressively and selectively expand the size of a pre-trained large language model (LLM) to improve model quality without end-to-end retraining.
  arXiv  Detail & Related papers  (2025-03-21T19:56:59Z)
• Graph-Augmented Reasoning: Evolving Step-by-Step Knowledge Graph Retrieval for LLM Reasoning [55.6623318085391]
  Recent large language model (LLM) reasoning suffers from limited domain knowledge, susceptibility to hallucinations, and constrained reasoning depth. This paper presents the first investigation into integrating step-wise knowledge graph retrieval with step-wise reasoning. We propose KG-RAR, a framework centered on process-oriented knowledge graph construction, a hierarchical retrieval strategy, and a universal post-retrieval processing and reward model.
  arXiv  Detail & Related papers  (2025-03-03T15:20:41Z)
• GAQAT: gradient-adaptive quantization-aware training for domain generalization [54.31450550793485]
  We propose a novel Gradient-Adaptive Quantization-Aware Training (GAQAT) framework for DG. Our approach begins by identifying the scale-gradient conflict problem in low-precision quantization. Extensive experiments validate the effectiveness of the proposed GAQAT framework.
  arXiv  Detail & Related papers  (2024-12-07T06:07:21Z)
• Art and Science of Quantizing Large-Scale Models: A Comprehensive Overview [4.166341398835636]
  We discuss the necessity and impact of model size growth, highlighting the performance benefits as well as the computational challenges and environmental considerations. We delve into various quantization techniques, including both post-training quantization (PTQ) and quantization-aware training (QAT) We examine how these methods address issues like outliers, importance weighting, and activation quantization, ultimately contributing to more sustainable and accessible deployment of large-scale models.
  arXiv  Detail & Related papers  (2024-09-18T02:35:00Z)
• Exploring the Trade-Offs: Quantization Methods, Task Difficulty, and Model Size in Large Language Models From Edge to Giant [11.832907585157638]
  Quantization has gained attention as a promising solution for the cost-effective deployment of large and small language models. We conduct a comprehensive evaluation of instruction-tuned models spanning 1B to 405B parameters, applying four quantization methods across 13 datasets.
  arXiv  Detail & Related papers  (2024-09-17T10:31:37Z)
• Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners [17.43650511873449]
  Large Language Models (LLMs) showcase remarkable performance and robust deductive capabilities, yet their expansive size complicates deployment and raises environmental concerns due to substantial resource consumption. We have developed innovative methods that enhance the performance of quantized LLMs, particularly in low-bit settings. Our methods consistently deliver state-of-the-art results across various quantization scenarios and offer deep theoretical insights into the quantization process, elucidating the potential of quantized models for widespread application.
  arXiv  Detail & Related papers  (2024-07-22T09:45:16Z)
• WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More [55.0856305773081]
  Large Language Models (LLMs) face significant deployment challenges due to their substantial memory requirements and the computational demands of auto-regressive text generation process. This paper addresses these challenges by focusing on the quantization of LLMs, a technique that reduces memory consumption by converting model parameters and activations into low-bit integers.
  arXiv  Detail & Related papers  (2024-02-19T11:33:21Z)
• Effect of Weight Quantization on Learning Models by Typical Case Analysis [6.9060054915724]
  The recent surge in data analysis scale has significantly increased computational resource requirements. Quantization is vital for deploying large models on devices with limited computational resources.
  arXiv  Detail & Related papers  (2024-01-30T18:58:46Z)
• Efficient Neural PDE-Solvers using Quantization Aware Training [71.0934372968972]
  We show that quantization can successfully lower the computational cost of inference while maintaining performance. Our results on four standard PDE datasets and three network architectures show that quantization-aware training works across settings and three orders of FLOPs magnitudes.
  arXiv  Detail & Related papers  (2023-08-14T09:21:19Z)
• Do Emergent Abilities Exist in Quantized Large Language Models: An Empirical Study [90.34226812493083]
  This work aims to investigate the impact of quantization on emphemergent abilities, which are important characteristics that distinguish LLMs from small language models. Our empirical experiments show that these emergent abilities still exist in 4-bit quantization models, while 2-bit models encounter severe performance degradation. To improve the performance of low-bit models, we conduct two special experiments: (1) fine-gained impact analysis that studies which components (or substructures) are more sensitive to quantization, and (2) performance compensation through model fine-tuning.
  arXiv  Detail & Related papers  (2023-07-16T15:11:01Z)
• End-to-end resource analysis for quantum interior point methods and portfolio optimization [63.4863637315163]
  We provide a complete quantum circuit-level description of the algorithm from problem input to problem output. We report the number of logical qubits and the quantity/depth of non-Clifford T-gates needed to run the algorithm.
  arXiv  Detail & Related papers  (2022-11-22T18:54:48Z)
• SQuAT: Sharpness- and Quantization-Aware Training for BERT [43.049102196902844]
  We propose sharpness- and quantization-aware training (SQuAT) Our method can consistently outperform state-of-the-art quantized BERT models under 2, 3, and 4-bit settings by 1%. Our experiments on empirical measurement of sharpness also suggest that our method would lead to flatter minima compared to other quantization methods.
  arXiv  Detail & Related papers  (2022-10-13T16:52:19Z)
• A Survey of Quantization Methods for Efficient Neural Network Inference [75.55159744950859]
  quantization is the problem of distributing continuous real-valued numbers over a fixed discrete set of numbers to minimize the number of bits required. It has come to the forefront in recent years due to the remarkable performance of Neural Network models in computer vision, natural language processing, and related areas. Moving from floating-point representations to low-precision fixed integer values represented in four bits or less holds the potential to reduce the memory footprint and latency by a factor of 16x.
  arXiv  Detail & Related papers  (2021-03-25T06:57:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.