SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization

• URL: http://arxiv.org/abs/2603.08185v1
• Date: Mon, 09 Mar 2026 10:04:12 GMT
• Title: SERQ: Saliency-Aware Low-Rank Error Reconstruction for LLM Quantization
• Abstract summary: Post-training quantization (PTQ) has emerged as a prevailing technique for deploying large language models efficiently.<n>We propose SERQ, a saliency-aware error reconstruction method for low-bit LLM inference that employs a single low-rank compensation matrix.
• Score: 7.372706701787234
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Post-training quantization (PTQ) has emerged as a prevailing technique for deploying large language models (LLMs) efficiently in terms of both memory and computation, across edge devices and server platforms. Existing PTQ methods primarily aim to reduce precision in weights and activations by mitigating quantization errors caused by channel-wise outlier activations (e.g., pre-quantization scaling, online transformations, or low-rank error reconstruction). Among these approaches, error reconstruction with low-rank adaptation (LoRA) has proven particularly effective, as it introduces a lightweight auxiliary computation path without requiring heavy optimization or additional online layers. However, prior studies reveal severe accuracy degradation under W4A4 settings, and conventional low-rank adaptations rely on two sequential factors, necessitating intermediate quantization during inference and thereby limiting low-precision efficiency. In this work, we propose SERQ, a saliency-aware error reconstruction method for low-bit LLM inference that employs a single low-rank compensation matrix. SERQ preserves efficient 4-bit matrix multiplication in linear layers by jointly mitigating quantization errors arising from both activation and weight saliency through three stages: (1) static activation flattening, (2) saliency-aware error reconstruction, and (3) offline weight permutation. The method incurs additional computation only for low-rank error reconstruction via a single decomposition, while all other operations are performed offline, thereby keeping latency overhead minimal. Empirically, SERQ outperforms prior error reconstruction methods under both W4A8 and W4A4 settings, and achieves higher accuracy than state-of-the-art rotation-based W4A4 approaches, while substantially reducing calibration complexity.

Related papers

• Low-Rank Ternary Adaptation for Fine-Tuning Transformers [54.3230438745856]
  Ternary transformers offer extreme memory and compute efficiency.<n>Existing low-bit LoRA-based methods cannot directly fine-tune ternary weights.<n>We propose ternary multiplicative adaptation, which represents discrete updates of ternary weights.
  arXiv  Detail & Related papers  (2026-08-25T12:15:56Z)
• QUASAR: Lowering the Loss Floor of Quantization-Aware Training with Loss-Aware Reconstruction [29.200349824129102]
  QUASAR continuously performs loss-aware reconstruction in the training loop to lower the loss floor and improve the resulting low-bit model.<n>It achieves the lowest held-out KL divergence among competitive QAT methods at 2, 3, and 4 bits.<n>It improves average accuracy across eight tasks by 3.5-4.3 percentage points over strong QAT and PTQ baselines.
  arXiv  Detail & Related papers  (2026-08-14T05:29:58Z)
• TWLA: Achieving Ternary Weights and Low-Bit Activations for LLMs via Post-Training Quantization [15.401450705732232]
  Ternarization has emerged as a promising compression technique.<n>Existing methods struggle with heavy-tailed activation distributions.<n>We propose TWLA, a framework that achieves 1.58-bit weight compression and 4-bit activation quantization.
  arXiv  Detail & Related papers  (2026-06-11T08:37:20Z)
• OffQ: Taming Structured Outliers in LLM Quantization by Offsetting [46.325071880831445]
  OffQ mitigates activation outliers in low-bit quantization through a novel offsetting mechanism.<n>OffQ consistently improves model accuracy while preserving low-bit efficiency.
  arXiv  Detail & Related papers  (2026-06-05T10:11:34Z)
• LoRaQ: Optimized Low Rank Approximation for 4-bit Quantization [39.25504627568235]
  Post-training quantization (PTQ) is essential for deploying large diffusion transformers on resource-constrained hardware.<n>LoRaQ (Low-Rank Approximated Quantization) is a simple, data-free calibration approach that optimize quantization error compensation.<n>We demonstrate that LoRaQ outperforms the state-of-the-art methods in their native implementations on Pixart-$$ and SANA.
  arXiv  Detail & Related papers  (2026-04-20T11:37:10Z)
• CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training [73.46600457802693]
  We introduce a new method that counteracts the loss induced by quantization.<n>CAGE significantly improves upon the state-of-theart methods in terms of accuracy, for similar computational cost.<n>For QAT pre-training of Llama models, CAGE matches the accuracy achieved at 4-bits (W4A4) with the prior best method.
  arXiv  Detail & Related papers  (2025-10-21T16:33:57Z)
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost [53.25965863436039]
  Quantization-aware training (QAT) provides a more principled solution, but its reliance on backpropagation incurs prohibitive memory costs.<n>We propose ZeroQAT, a zeroth-order optimization-based QAT framework that supports both weight and activation quantization.<n>Experiments show that ZeroQAT consistently outperforms representative PTQ and QAT baselines while requiring significantly less memory.
  arXiv  Detail & Related papers  (2025-08-21T01:18:27Z)
• GAQAT: gradient-adaptive quantization-aware training for domain generalization [54.31450550793485]
  We propose a novel Gradient-Adaptive Quantization-Aware Training (GAQAT) framework for DG.<n>Our approach begins by identifying the scale-gradient conflict problem in low-precision quantization.<n>Extensive experiments validate the effectiveness of the proposed GAQAT framework.
  arXiv  Detail & Related papers  (2024-12-07T06:07:21Z)
• EoRA: Fine-tuning-free Compensation for Compressed LLM with Eigenspace Low-Rank Approximation [84.70637613266835]
  EoRA is a fine-tuning-free method that augments compressed Large Language Models with low-rank matrices.<n>EoRA consistently outperforms prior training-free low rank methods in recovering the accuracy of compressed LLMs.
  arXiv  Detail & Related papers  (2024-10-28T17:59:03Z)
• Towards Accurate Post-Training Quantization of Vision Transformers via Error Reduction [48.740630807085566]
  Post-training quantization (PTQ) for vision transformers (ViTs) has received increasing attention from both academic and industrial communities.<n>Current methods fail to account for the complex interactions between quantized weights and activations, resulting in significant quantization errors and suboptimal performance.<n>This paper presents ERQ, an innovative two-step PTQ method specifically crafted to reduce quantization errors arising from activation and weight quantization sequentially.
  arXiv  Detail & Related papers  (2024-07-09T12:06:03Z)
• AffineQuant: Affine Transformation Quantization for Large Language Models [58.45460102764]
  Post-Training Quantization (PTQ) has emerged as a subject of considerable interest due to its compression efficiency and cost-effectiveness in the context of training. Existing PTQ methods for Large-scale Language Models (LLMs) limit the optimization scope to scaling transformations between pre- and post-quantization weights. In this paper, we advocate for the direct optimization using equivalent Affine transformations in PTQ (AffineQuant)
  arXiv  Detail & Related papers  (2024-03-19T08:40:21Z)
• L4Q: Parameter Efficient Quantization-Aware Fine-Tuning on Large Language Models [5.304907804008533]
  We propose L4Q, a method that integrates Quantization-Aware Training (QAT) with Low-Rank Adaptation (LoRA)<n>By employing a memory-optimized layer design, L4Q significantly reduces QAT's memory overhead, making its training cost comparable to LoRA.<n>Our experiments demonstrate that this combined approach to quantization and fine-tuning achieves superior accuracy.
  arXiv  Detail & Related papers  (2024-02-07T14:35:05Z)
• LQER: Low-Rank Quantization Error Reconstruction for LLMs [13.205129808742862]
  We introduce Low-rank Quantization Error Reduction (LQER), which combines quantization and low-rank approximation to recover the model capability. Unlike existing methods, the computation pattern of LQER eliminates the need for specialized Scatter and Gather processes. Our W4A8 LLMs achieve near-lossless performance on six popular downstream tasks, while using 1.36$times$ fewer hardware resources than the leading state-of-the-art method.
  arXiv  Detail & Related papers  (2024-02-04T10:59:52Z)
• CBQ: Cross-Block Quantization for Large Language Models [66.82132832702895]
  Post-training quantization (PTQ) has played a key role in compressing large language models (LLMs) with ultra-low costs.<n>We propose CBQ, a cross-block reconstruction-based PTQ method for LLMs.<n> CBQ employs a cross-block dependency using a reconstruction scheme, establishing long-range dependencies across multiple blocks to minimize error accumulation.
  arXiv  Detail & Related papers  (2023-12-13T07:56:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.