Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method

• URL: http://arxiv.org/abs/2507.18073v1
• Date: Thu, 24 Jul 2025 03:55:19 GMT
• Title: Squeeze10-LLM: Squeezing LLMs' Weights by 10 Times via a Staged Mixed-Precision Quantization Method
• Abstract summary: We propose Squeeze10-LLM to "squeezing" 16-bit language models' weights by 10 times.<n>It achieves an average of 1.6 bits per weight by quantizing 80% of the weights to 1 bit and 20% to 4 bits.<n> Experiments on LLaMA and LLaMA2 show that Squeeze10-LLM achieves state-of-the-art performance for sub-2bit weight-only quantization.
• Score: 37.70474075872739
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deploying large language models (LLMs) is challenging due to their massive parameters and high computational costs. Ultra low-bit quantization can significantly reduce storage and accelerate inference, but extreme compression (i.e., mean bit-width <= 2) often leads to severe performance degradation. To address this, we propose Squeeze10-LLM, effectively "squeezing" 16-bit LLMs' weights by 10 times. Specifically, Squeeze10-LLM is a staged mixed-precision post-training quantization (PTQ) framework and achieves an average of 1.6 bits per weight by quantizing 80% of the weights to 1 bit and 20% to 4 bits. We introduce Squeeze10LLM with two key innovations: Post-Binarization Activation Robustness (PBAR) and Full Information Activation Supervision (FIAS). PBAR is a refined weight significance metric that accounts for the impact of quantization on activations, improving accuracy in low-bit settings. FIAS is a strategy that preserves full activation information during quantization to mitigate cumulative error propagation across layers. Experiments on LLaMA and LLaMA2 show that Squeeze10-LLM achieves state-of-the-art performance for sub-2bit weight-only quantization, improving average accuracy from 43% to 56% on six zero-shot classification tasks--a significant boost over existing PTQ methods. Our code will be released upon publication.

Related papers

• Minimizing the Hidden Cost of Scales: Graph-Guided Ultra-Low-Bit Quantization for Large Language Models [50.16014098038291]
  Post-training quantization (PTQ) is critical for the efficient deployment of large language models (LLMs)<n>We propose SAGE-PTQ, a novel ultra-low-bit quantization framework for LLMs that minimizes hidden scaling cost.<n>On LLaMA-3B, SAGE-PTQ achieves 6.74 WikiText2 perplexity, compared to 55.8 for BiLLM, while using less than 50% of BiLLM's GPU memory.
  arXiv  Detail & Related papers  (2026-06-03T20:51:52Z)
• BWLA: Breaking the Barrier of W1AX Post-Training Quantization for LLMs [10.07268309735318]
  Large language models (LLMs) have driven major progress in NLP, yet their substantial memory and compute demands still hinder practical deployment.<n>We propose BWLA, the first post-training quantization framework that preserves high accuracy while achieving 1-bit weight quantization.<n>On Qwen3-32B, BWLA reaches a Wikitext2 perplexity of 11.92 under 6-bit activations, improves five zero-shot tasks by more than 70%, and delivers 3.26 times inference speedup.
  arXiv  Detail & Related papers  (2026-05-01T05:42:57Z)
• D$^2$Quant: Accurate Low-bit Post-Training Weight Quantization for LLMs [33.883527341335856]
  Weight-only post-training quantization (PTQ) is appealing as it reduces memory usage and enables practical speedup without low-bit operators or specialized hardware.<n> accuracy often degrades significantly in weight-only PTQ at sub-4-bit precision.<n>We propose D$2$Quant, a novel weight-only PTQ framework that improves quantization from both the weight and activation perspectives.
  arXiv  Detail & Related papers  (2026-01-30T05:49:48Z)
• Rethinking Output Alignment For 1-bit Post-Training Quantization of Large Language Models [41.677469535447024]
  Large Language Models (LLMs) deliver strong performance across a wide range of NLP tasks, but their massive sizes hinder deployment on resource-constrained devices.<n>Post-training quantization (PTQ) is widely adopted for its efficiency, as it requires no retraining and only a small dataset for calibration.<n>Recent advances for post-training quantization have demonstrated that even sub-4-bit methods can maintain most of the original model performance.
  arXiv  Detail & Related papers  (2025-12-25T12:39:36Z)
• SignRoundV2: Closing the Performance Gap in Extremely Low-Bit Post-Training Quantization for LLMs [4.946856266233001]
  SignRoundV2 is a post-training quantization framework that is highly effective even without mixed-precision.<n>Our method sustains competitive accuracy for Large Language Models, achieving production-grade performance with about 1 percent variance at 4-5 bits.
  arXiv  Detail & Related papers  (2025-12-04T12:35:10Z)
• LittleBit: Ultra Low-Bit Quantization via Latent Factorization [7.1508066212157715]
  Large language models (LLMs) often face challenges from substantial memory and computational costs.<n>This paper introduces LittleBit, a novel method for extreme LLM compression.<n>It achieves nearly 31$times$ memory reduction, e.g., Llama2-13B to under 0.9 GB.
  arXiv  Detail & Related papers  (2025-05-30T06:43:03Z)
• LoTA-QAF: Lossless Ternary Adaptation for Quantization-Aware Fine-Tuning [50.89500210372827]
  Quantization and fine-tuning are crucial for deploying large language models (LLMs) on resource-constrained edge devices.<n>LoTA-QAF is a novel fine-tuning method specifically designed for quantized LLMs.<n>On the MMLU benchmark, our method effectively recovers performance for quantized models, surpassing 16-bit LoRA by up to 5.14%.
  arXiv  Detail & Related papers  (2025-05-24T14:47:28Z)
• PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models [64.84734437930362]
  Large Language Models (LLMs) suffer severe performance degradation when facing extremely low-bit (sub 2-bit) quantization.<n>We propose an extremely low-bit PTQ method called PTQ1.61, which enables weight quantization to 1.61-bit for the first time.<n>Experiments indicate our PTQ1.61 achieves state-of-the-art performance in extremely low-bit quantization.
  arXiv  Detail & Related papers  (2025-02-18T08:04:58Z)
• ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models [9.444063879246242]
  We introduce a novel arbitrary-bit quantization algorithm and inference framework, ABQ-LLM.<n>It achieves superior performance across various quantization settings and enables efficient arbitrary-precision quantized inference on the GPU.
  arXiv  Detail & Related papers  (2024-08-16T06:39:08Z)
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models [63.118592279833656]
  Post-training quantization (PTQ) is an effective technique for compressing large language models (LLMs)<n>We propose SliM-LLM, a salience-driven mixed-precision quantization framework that allocates bit-widths at the group-wise.<n> Experiments show that SliM-LLM achieves superior performance across various LLMs at low bit-widths.
  arXiv  Detail & Related papers  (2024-05-23T16:21:48Z)
• OneBit: Towards Extremely Low-bit Large Language Models [66.29839811207617]
  This paper boldly quantizes the weight matrices of LLMs to 1-bit, paving the way for the extremely low bit-width deployment of LLMs.<n>Experiments indicate that OneBit achieves good performance (at least 81% of the non-quantized performance on LLaMA models) with robust training processes.
  arXiv  Detail & Related papers  (2024-02-17T14:26:57Z)
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs [53.31402059062365]
  BiLLM is a groundbreaking 1-bit post-training quantization scheme tailored for pretrained large language models. It achieves for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families.
  arXiv  Detail & Related papers  (2024-02-06T09:26:34Z)
• QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Models [44.515165695546614]
  Quantization-Aware Training (QAT) offers a solution, but its extensive training costs make Post-Training Quantization (PTQ) a more practical approach for Large Language Models (LLMs) We propose QLLM, an accurate and efficient low-bitwidth PTQ method designed for LLMs.
  arXiv  Detail & Related papers  (2023-10-12T05:25:49Z)
• Norm Tweaking: High-performance Low-bit Quantization of Large Language Models [21.855106896725598]
  We introduce a technique called norm tweaking, which can be used as a plugin in current PTQ methods to achieve high precision. Our method demonstrates significant improvements in both weight-only quantization and joint quantization of weights and activations. Our simple and effective approach makes it more practical for real-world applications.
  arXiv  Detail & Related papers  (2023-09-06T06:51:15Z)
• OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Models [57.27101446992148]
  Large language models (LLMs) have revolutionized natural language processing tasks. Recent post-training quantization (PTQ) methods are effective in reducing memory footprint and improving the computational efficiency of LLM. We introduce an Omnidirectionally calibrated Quantization technique for LLMs, which achieves good performance in diverse quantization settings.
  arXiv  Detail & Related papers  (2023-08-25T02:28:35Z)
• AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration [54.692405042065815]
  We propose Activation-aware Weight Quantization (AWQ), a hardware-friendly approach for LLM low-bit weight-only quantization. AWQ protects only 1% salient weights and achieves excellent quantization performance for instruction-tuned LMs and, for the first time, multi-modal LMs. We also implement TinyChat, an efficient and flexible inference framework tailored for 4-bit on-device LLM/VLMs.
  arXiv  Detail & Related papers  (2023-06-01T17:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.