Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition

• URL: http://arxiv.org/abs/2506.02077v1
• Date: Mon, 02 Jun 2025 09:15:13 GMT
• Title: Assigning Distinct Roles to Quantized and Low-Rank Matrices Toward Optimal Weight Decomposition
• Authors: Yoonjun Cho, Soeun Kim, Dongjae Jeon, Kyelim Lee, Beomsoo Lee, Albert No,
• Abstract summary: We introduce Outlier-Driven Low-Rank Initialization (ODLRI) which assigns low-rank components the specific role of capturing activation-sensitive weights.<n>Experiments on Llama2 (7B, 13B, 70B), Llama3-8B, and Mistral-7B demonstrate that ODLRI consistently reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy in low-bit settings.
• Score: 4.119890956388359
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Decomposing weight matrices into quantization and low-rank components ($\mathbf{W} \approx \mathbf{Q} + \mathbf{L}\mathbf{R}$) is a widely used technique for compressing large language models (LLMs). Existing joint optimization methods iteratively alternate between quantization and low-rank approximation. However, these methods tend to prioritize one component at the expense of the other, resulting in suboptimal decompositions that fail to leverage each component's unique strengths. In this work, we introduce Outlier-Driven Low-Rank Initialization (ODLRI), which assigns low-rank components the specific role of capturing activation-sensitive weights. This structured decomposition mitigates outliers' negative impact on quantization, enabling more effective balance between quantization and low-rank approximation. Experiments on Llama2 (7B, 13B, 70B), Llama3-8B, and Mistral-7B demonstrate that incorporating ODLRI into the joint optimization framework consistently reduces activation-aware error, minimizes quantization scale, and improves perplexity and zero-shot accuracy in low-bit settings.

Related papers

• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• IMPACT: Importance-Aware Activation Space Reconstruction [5.487612141214714]
  Large language models (LLMs) achieve strong performance across many domains but are difficult to deploy in resource-constrained settings due to their size.<n>We propose IMPACT, a principled framework for importance-aware activation reconstruction that links model compression decisions to their impact on model behavior.<n>Experiments across diverse models and tasks show that IMPACT achieves up to 48.6% greater model size reduction with accuracy comparable to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-07-04T22:26:33Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE)<n>RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers.<n>Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• HASSLE-free: A unified Framework for Sparse plus Low-Rank Matrix Decomposition for LLMs [15.575498324678373]
  A promising compression scheme is to decompose foundation models' dense weights into a sum of sparse plus low-rank matrices.<n>In this paper, we design a unified framework coined HASSLE-free for (semi-structured) sparse plus low-rank matrix decomposition.
  arXiv  Detail & Related papers  (2025-02-02T20:23:32Z)
• ASER: Activation Smoothing and Error Reconstruction for Large Language Model Quantization [18.017182472532415]
  ASER is an algorithm consisting of low-rank compensation for quantization error with LoRA-style matrices constructed by whitening SVD.<n>ASER is capable of quantizing typical outliers to low-bit ones, particularly preserving accuracy even in W4A8 per-channel setup.
  arXiv  Detail & Related papers  (2024-11-12T12:52:04Z)
• Compensate Quantization Errors+: Quantized Models Are Inquisitive Learners [51.32182730502002]
  We introduce Singular-value Diagonal Expansion to refine weight distributions to achieve better quantization alignment.<n>Our plug-and-play weight-quantization methods demonstrate substantial performance improvements over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2024-07-22T09:45:16Z)
• From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applications [85.17672240603011]
  We study the non-uniform low-rank properties of weight matrices in Large Language Models.<n>We present Weight Low-Rank Projection (WeLore) that unifies weight compression and memory-efficient fine-tuning into one.
  arXiv  Detail & Related papers  (2024-07-15T21:05:20Z)
• BoA: Attention-aware Post-training Quantization without Backpropagation [11.096116957844014]
  Post-training quantization (PTQ) is a promising solution for deploying large language models (LLMs) on resource-constrained devices.<n>We introduce a novel backpropagation-free PTQ algorithm that optimize integer weights by considering inter-layer dependencies.
  arXiv  Detail & Related papers  (2024-06-19T11:53:21Z)
• SliM-LLM: Salience-Driven Mixed-Precision Quantization for Large Language Models [63.118592279833656]
  Post-training quantization (PTQ) is an effective technique for compressing large language models (LLMs)<n>We propose SliM-LLM, a salience-driven mixed-precision quantization framework that allocates bit-widths at the group-wise.<n> Experiments show that SliM-LLM achieves superior performance across various LLMs at low bit-widths.
  arXiv  Detail & Related papers  (2024-05-23T16:21:48Z)
• LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning [66.85589263870702]
  Our approach uses an iterative algorithm to decompose each pretrained matrix into a high-precision low-rank component and a memory-efficient quantized component. Experiments on finetuning RoBERTa and LLaMA-2 demonstrate that our low-rank plus quantized matrix decomposition approach (LQ-LoRA) outperforms strong QLoRA and GPTQ-LoRA baselines.
  arXiv  Detail & Related papers  (2023-11-20T18:57:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.