Mixed-Precision Graph Neural Quantization for Low Bit Large Language Models

• URL: http://arxiv.org/abs/2501.18154v1
• Date: Thu, 30 Jan 2025 05:39:01 GMT
• Title: Mixed-Precision Graph Neural Quantization for Low Bit Large Language Models
• Authors: Wanlong Liu, Yichen Xiao, Dingyi Zeng, Hongyang Zhao, Wenyu Chen, Malu Zhang,
• Abstract summary: Post-Training Quantization (PTQ) is pivotal for deploying large language models within resource-limited settings. We introduce a Mixed-precision Graph Neural PTQ (MG-PTQ) approach, employing a graph neural network (GNN) module to capture dependencies among weights. Our method more effectively captures dependencies among target weights, leading to a more accurate assessment of weight importance.
• Score: 13.709080134204326
• License:
• Abstract: Post-Training Quantization (PTQ) is pivotal for deploying large language models (LLMs) within resource-limited settings by significantly reducing resource demands. However, existing PTQ strategies underperform at low bit levels < 3 bits due to the significant difference between the quantized and original weights. To enhance the quantization performance at low bit widths, we introduce a Mixed-precision Graph Neural PTQ (MG-PTQ) approach, employing a graph neural network (GNN) module to capture dependencies among weights and adaptively assign quantization bit-widths. Through the information propagation of the GNN module, our method more effectively captures dependencies among target weights, leading to a more accurate assessment of weight importance and optimized allocation of quantization strategies. Extensive experiments on the WikiText2 and C4 datasets demonstrate that our MG-PTQ method outperforms previous state-of-the-art PTQ method GPTQ, setting new benchmarks for quantization performance under low-bit conditions.

Related papers

• PTQ1.61: Push the Real Limit of Extremely Low-Bit Post-Training Quantization Methods for Large Language Models [64.84734437930362]
  Large Language Models (LLMs) suffer severe performance degradation when facing extremely low-bit (sub 2-bit) quantization. We propose an extremely low-bit PTQ method called PTQ1.61, which enables weight quantization to 1.61-bit for the first time. Experiments indicate our PTQ1.61 achieves state-of-the-art performance in extremely low-bit quantization.
  arXiv  Detail & Related papers  (2025-02-18T08:04:58Z)
• GWQ: Gradient-Aware Weight Quantization for Large Language Models [63.89099994367657]
  Large language models (LLMs) show impressive performance in solving complex language tasks. LLMs to low bits can enable them to run on resource-constrained devices, often leading to performance degradation. We propose gradient-aware weight quantization (GWQ), the first quantization approach for low-bit weight quantization.
  arXiv  Detail & Related papers  (2024-10-30T11:16:04Z)
• QT-DoG: Quantization-aware Training for Domain Generalization [58.439816306817306]
  We propose Quantization-aware Training for Domain Generalization (QT-DoG) QT-DoG exploits quantization as an implicit regularizer by inducing noise in model weights. We demonstrate that QT-DoG generalizes across various datasets, architectures, and quantization algorithms.
  arXiv  Detail & Related papers  (2024-10-08T13:21:48Z)
• PB-LLM: Partially Binarized Large Language Models [14.244537605866864]
  This paper explores network binarization, compressing model weights to a single bit, specifically for Large Language Models (LLMs) compression. We propose a novel approach, Partially-Binarized LLM (PB-LLM), which can achieve extreme low-bit quantization while maintaining the linguistic reasoning capacity of quantized LLMs.
  arXiv  Detail & Related papers  (2023-09-29T14:35:27Z)
• EPTQ: Enhanced Post-Training Quantization via Hessian-guided Network-wise Optimization [3.3998740964877463]
  Quantization is a key method for deploying deep neural networks on edge devices with limited memory and computation resources. We propose a new method for enhanced Post-Training Quantization (EPTQ) that employs a network-wise quantization optimization process.
  arXiv  Detail & Related papers  (2023-09-20T10:50:28Z)
• Norm Tweaking: High-performance Low-bit Quantization of Large Language Models [21.855106896725598]
  We introduce a technique called norm tweaking, which can be used as a plugin in current PTQ methods to achieve high precision. Our method demonstrates significant improvements in both weight-only quantization and joint quantization of weights and activations. Our simple and effective approach makes it more practical for real-world applications.
  arXiv  Detail & Related papers  (2023-09-06T06:51:15Z)
• Quantization Aware Factorization for Deep Neural Network Compression [20.04951101799232]
  decomposition of convolutional and fully-connected layers is an effective way to reduce parameters and FLOP in neural networks. A conventional post-training quantization approach applied to networks with weights yields a drop in accuracy. This motivated us to develop an algorithm that finds decomposed approximation directly with quantized factors.
  arXiv  Detail & Related papers  (2023-08-08T21:38:02Z)
• BiTAT: Neural Network Binarization with Task-dependent Aggregated Transformation [116.26521375592759]
  Quantization aims to transform high-precision weights and activations of a given neural network into low-precision weights/activations for reduced memory usage and computation. Extreme quantization (1-bit weight/1-bit activations) of compactly-designed backbone architectures results in severe performance degeneration. This paper proposes a novel Quantization-Aware Training (QAT) method that can effectively alleviate performance degeneration.
  arXiv  Detail & Related papers  (2022-07-04T13:25:49Z)
• Cluster-Promoting Quantization with Bit-Drop for Minimizing Network Quantization Loss [61.26793005355441]
  Cluster-Promoting Quantization (CPQ) finds the optimal quantization grids for neural networks. DropBits is a new bit-drop technique that revises the standard dropout regularization to randomly drop bits instead of neurons. We experimentally validate our method on various benchmark datasets and network architectures.
  arXiv  Detail & Related papers  (2021-09-05T15:15:07Z)
• Gradient $\ell_1$ Regularization for Quantization Robustness [70.39776106458858]
  We derive a simple regularization scheme that improves robustness against post-training quantization. By training quantization-ready networks, our approach enables storing a single set of weights that can be quantized on-demand to different bit-widths.
  arXiv  Detail & Related papers  (2020-02-18T12:31:34Z)
• SQWA: Stochastic Quantized Weight Averaging for Improving the Generalization Capability of Low-Precision Deep Neural Networks [29.187848543158992]
  We present a new quantized neural network optimization approach, quantized weight averaging (SQWA) The proposed approach includes floating-point model training, direct quantization of weights, capturing multiple low-precision models, averaging the captured models, and fine-tuning it with low-learning rates. With SQWA training, we achieved state-of-the-art results for 2-bit QDNNs on CIFAR-100 and ImageNet datasets.
  arXiv  Detail & Related papers  (2020-02-02T07:02:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.