CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs

• URL: http://arxiv.org/abs/2606.26650v1
• Date: Thu, 25 Jun 2026 06:24:02 GMT
• Title: CAT-Q: Cost-efficient and Accurate Ternary Quantization for LLMs
• Abstract summary: We present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs.<n>For pre-trained LLMs with 1.7B to 8B parameters, CAT-Q can efficiently quantize them into ternary models using only 512 calibration samples.<n>We show for the first time that CAT-Q can quantize much larger pre-trained LLMs having 14B to 235B parameters into leading ternary models within just 8 to 60 hours.
• Score: 13.903045659324187
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we present CAT-Q, Cost-efficient and Accurate Ternary Quantization, for compressing and accelerating LLMs. Unlike existing state-of-the-art ternary quantization methods that rely on data-intensive and costly quantization-aware training to mitigate severe performance degradation, CAT-Q is a simple yet effective post-training quantization scheme that is readily applicable to LLMs with diverse architectures and model sizes. It has two key components, learnable modulation (LM) and softened ternarization (ST), which are coupled from an optimization perspective. LM leverages a composition of learnable factors to modulate the distribution of pre-trained high-precision weights and the ternary threshold, making them less sensitive to ternarization. ST further introduces a differentiable transition function to guide the ternarization process toward stable convergence. We show that, for pre-trained LLMs with 1.7B to 8B parameters, CAT-Q can efficiently quantize them into ternary models using only 512 calibration samples, while achieving superior performance than the seminal BitNet 1.58-bit v1 and v2 families (with 1.3B to 7B parameters) trained with 100B tokens, yielding about a 100,000X reduction in training tokens. Moreover, we show for the first time that CAT-Q can quantize much larger pre-trained LLMs having 14B to 235B parameters into leading ternary models within just 8 to 60 hours on 8 A100-80GB GPUs. Code is available at https://github.com/IntelChina-AI/BitTern.

Related papers

• Attend to Your Own Thoughts: Breaking the Barrier for Post-Training Quantization of Reasoning LLMs through the Lens of 1.58-Bit Quantization [13.903045659324187]
  We propose ScaleQ-1.58, a scalable ternary post-training quantization framework for reasoning LLMs.<n>It is formed by simply integrating AYOT with CAT-Q, which demonstrates several scaling properties.<n>It generalizes well to both dense and MoE architectures, with performance improving as model scale increases.
  arXiv  Detail & Related papers  (2026-08-02T08:22:43Z)
• TernaryLM: Memory-Efficient Language Modeling via Native 1-Bit Quantization with Adaptive Layer-wise Scaling [0.39287497907611874]
  We present TernaryLM, a 132M parameter transformer architecture that employs native 1-bit ternary quantization -1, 0, +1 during training.<n>Our results suggest that native 1-bit training is a promising direction for efficient neural language models.
  arXiv  Detail & Related papers  (2026-02-07T05:35:17Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE)<n>RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers.<n>Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• HALO: Hadamard-Assisted Lower-Precision Optimization for LLMs [48.55966021231297]
  We present HALO, a novel quantization-aware training approach for Transformers.<n>Our approach ensures that all large matrix multiplications during the forward and backward passes are executed in lower precision.<n>Applying to LLAMA-family models, HALO achieves near-full-precision-equivalent results during fine-tuning on various tasks.
  arXiv  Detail & Related papers  (2025-01-05T18:41:54Z)
• EfficientQAT: Efficient Quantization-Aware Training for Large Language Models [50.525259103219256]
  quantization-aware training (QAT) offers a solution by reducing memory consumption through low-bit representations with minimal accuracy loss.<n>We propose Efficient Quantization-Aware Training (EfficientQAT), a more feasible QAT algorithm.<n> EfficientQAT involves two consecutive phases: Block-wise training of all parameters (Block-AP) and end-to-end training of quantization parameters (E2E-QP)
  arXiv  Detail & Related papers  (2024-07-10T17:53:30Z)
• OneBit: Towards Extremely Low-bit Large Language Models [66.29839811207617]
  This paper boldly quantizes the weight matrices of LLMs to 1-bit, paving the way for the extremely low bit-width deployment of LLMs.<n>Experiments indicate that OneBit achieves good performance (at least 81% of the non-quantized performance on LLaMA models) with robust training processes.
  arXiv  Detail & Related papers  (2024-02-17T14:26:57Z)
• BiLLM: Pushing the Limit of Post-Training Quantization for LLMs [53.31402059062365]
  BiLLM is a groundbreaking 1-bit post-training quantization scheme tailored for pretrained large language models. It achieves for the first time high-accuracy inference (e.g. 8.41 perplexity on LLaMA2-70B) with only 1.08-bit weights across various LLMs families.
  arXiv  Detail & Related papers  (2024-02-06T09:26:34Z)
• TEQ: Trainable Equivalent Transformation for Quantization of LLMs [1.0376648762140632]
  We present TEQ, a trainable equivalent transformation that preserves the FP32 precision of the model output while taking advantage of low-precision quantization. The training process is lightweight, requiring only 1K steps and fewer than 0.1 percent of the original model's trainable parameters.
  arXiv  Detail & Related papers  (2023-10-17T02:42:34Z)
• QFT: Quantized Full-parameter Tuning of LLMs with Affordable Resources [35.16907522675046]
  Large Language Models (LLMs) have showcased remarkable impacts across a wide spectrum of natural language processing tasks.<n>Fine-tuning these pretrained models on downstream datasets provides significant performance gains.<n>This process typically requires a large number of expensive, high-end GPU.<n>We propose QFT, a Quantized Full- parameter Tuning framework that quantizes and stores all training states.
  arXiv  Detail & Related papers  (2023-10-11T02:47:40Z)
• Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognition [67.95996816744251]
  State-of-the-art language models (LMs) represented by long-short term memory recurrent neural networks (LSTM-RNNs) and Transformers are becoming increasingly complex and expensive for practical applications. Current quantization methods are based on uniform precision and fail to account for the varying performance sensitivity at different parts of LMs to quantization errors. Novel mixed precision neural network LM quantization methods are proposed in this paper.
  arXiv  Detail & Related papers  (2021-11-29T12:24:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.