Intrinsic Structure as a Proxy for Saliency: SVD-Based Weight Preservation for Mixed-Precision Quantization in Large Language Models

• URL: http://arxiv.org/abs/2512.01343v2
• Date: Tue, 02 Dec 2025 05:26:59 GMT
• Title: Intrinsic Structure as a Proxy for Saliency: SVD-Based Weight Preservation for Mixed-Precision Quantization in Large Language Models
• Authors: Shashank Landge, Abhishek Patil, Tejas kamble, Bhushan Buddhivant, Priyanka Joshi,
• Abstract summary: Post-Training Quantization (PTQ) addresses this by reducing the precision of model weights, typically to 4-bit or lower.<n>Current state-of-the-art methods rely on calibration data to identify salient weights.<n>We propose a data-free, structure-aware hypothesis: that the weights identified as Principal Components via Singular Value Decomposition (SVD) are intrinsically important to the model's downstream performance.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Large Language Models (LLMs) continue to scale in parameter count, deploying them on commodity hardware has become increasingly challenging. Post-Training Quantization (PTQ) addresses this by reducing the precision of model weights, typically to 4-bit or lower. However, uniform quantization often leads to significant performance degradation due to the presence of ``outlier features'' -- weights that, while few in number, are critical for maintaining model accuracy. Current state-of-the-art methods such as AWQ (Activation-aware Weight Quantization) and SpQR (Sparse Quantization Representations) rely on calibration data to identify these salient weights via activation magnitudes or Hessian sensitivity. In scenarios where data privacy is paramount or calibration data is unavailable, these methods are inapplicable. In this work, we propose a data-free, structure-aware hypothesis: that the weights identified as Principal Components via Singular Value Decomposition (SVD) are intrinsically important to the model's downstream performance. We introduce a novel selection heuristic that preserves the top-$k$ weights aligned with the principal components in FP32, while aggressively quantizing the residual weights. We compare our method against activation-aware (AWQ) and second-order (SpQR) methods across GLUE benchmarks (MRPC, RTE, QNLI) using a DistilBERT backbone. Our experiments reveal that structural importance is highly correlated with functional importance. On the challenging RTE task, our SVD-based method achieves an accuracy of 66.06\%, outperforming both AWQ (65.34\%) and SpQR (65.34\%) at high protection budgets, validating that intrinsic matrix structure can serve as a robust proxy for weight saliency without the need for forward passes or calibration data.

Related papers

• S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations [30.145244497712792]
  Activation outliers in large-scale transformer models pose a fundamental challenge to model quantization.<n>We propose Selective Spectral Decay ($S2D$), a geometrically-principled conditioning method that surgically regularizes only the weight components corresponding to the largest singular values.<n>We show that $S2D$ significantly reduces activation outliers and produces well-conditioned representations that are inherently quantization-friendly.
  arXiv  Detail & Related papers  (2026-02-16T03:41:06Z)
• SASQ: Static Activation Scaling for Quantization-Aware Training in Large Language Models [6.235887167172886]
  We propose SASQ: a lightweight QAT framework specifically tailored for activation quantization factors.<n>On LLaMA2-7B, it achieves 5.2% lower perplexity than QuaRot and 4.7% lower perplexity than the FP16 model on WikiText2.
  arXiv  Detail & Related papers  (2025-12-16T15:12:34Z)
• Extreme Model Compression with Structured Sparsity at Low Precision [10.976782748075067]
  Deep neural networks (DNNs) are used in many applications, but their large size and high computational cost make them hard to run on devices with limited resources.<n>Two widely used techniques to address this challenge are weight quantization, which lowers the precision of all weights, and structured sparsity, which removes unimportant weights while retaining the important ones at full precision.<n>We introduce SLOPE Structured Sparsity at Low Precision, a unified framework, to effectively combine structured sparsity and low-bit quantization in a principled way.
  arXiv  Detail & Related papers  (2025-11-11T15:37:55Z)
• Quantized Visual Geometry Grounded Transformer [67.15451442018258]
  This paper proposes the first Quantization framework for VGGTs, namely QuantVGGT.<n>We introduce Dual-Smoothed Fine-Grained Quantization, which integrates pre-global Hadamard rotation and post-local channel smoothing.<n>We also design Noise-Filtered Diverse Sampling, which filters outliers via deep-layer statistics.
  arXiv  Detail & Related papers  (2025-09-25T15:17:11Z)
• Quantization Meets Reasoning: Exploring and Mitigating Degradation of Low-Bit LLMs in Mathematical Reasoning [39.56908863102256]
  Low-bit post-training quantization impairs mathematical reasoning up to 69.81% in harder settings.<n>We address two deployment-critical questions with process-level precision.<n>In our settings, as few as 332 curated examples and 3--5 minutes of compute on a single GPU recover 4-bit weight math reasoning toward the full-precision baseline.
  arXiv  Detail & Related papers  (2025-05-16T12:11:40Z)
• Task-Circuit Quantization: Leveraging Knowledge Localization and Interpretability for Compression [55.323397702682506]
  Post-training quantization (PTQ) reduces a model's memory footprint by mapping full precision weights into low bit weights without costly retraining.<n>We develop a new mixed-precision PTQ approach, Task-Circuit Quantization (TaCQ), that draws parallels to automated circuit discovery.
  arXiv  Detail & Related papers  (2025-04-10T02:19:03Z)
• RSQ: Learning from Important Tokens Leads to Better Quantized LLMs [65.5558181902098]
  Layer-wise quantization is a key technique for efficiently compressing large models without expensive retraining.<n>We propose RSQ (Rotate, Scale, then Quantize), which applies rotations to the model to mitigate outliers.<n>We demonstrate that RSQ consistently outperforms baseline methods across multiple downstream tasks and three model families.
  arXiv  Detail & Related papers  (2025-03-03T18:46:33Z)
• Identifying Sensitive Weights via Post-quantization Integral [27.722950830077444]
  We propose Post-quantization Integral (PQI) to estimate posterior sensitivity in a fine-grained manner.<n>We also propose ReQuant, a simple yet powerful framework that mainly consists of two-and-Sparse detach components.<n>Results show that ReQuant boosts state-of-the-art post-training quantization methods, with a pronounced improvement of 2.66 perplexity gain on Llama 3.2 1B with QTIP.
  arXiv  Detail & Related papers  (2025-02-28T07:04:19Z)
• GWQ: Gradient-Aware Weight Quantization for Large Language Models [56.22507677736051]
  Large language models (LLMs) show impressive performance in solving complex language tasks.<n> compressing LLMs to low bits can enable to deploy on resource-constrained devices.<n>We propose gradient-aware weight quantization (GWQ), the first quantization approach for low-bit weight quantization.
  arXiv  Detail & Related papers  (2024-10-30T11:16:04Z)
• Data-free Weight Compress and Denoise for Large Language Models [96.68582094536032]
  We propose a novel approach termed Data-free Joint Rank-k Approximation for compressing the parameter matrices.<n>We achieve a model pruning of 80% parameters while retaining 93.43% of the original performance without any calibration data.
  arXiv  Detail & Related papers  (2024-02-26T05:51:47Z)
• Language model compression with weighted low-rank factorization [73.61874728240568]
  We introduce Fisher information to weigh the importance of parameters affecting the model prediction. We find that our resulting task accuracy is much closer to the original model's performance. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies.
  arXiv  Detail & Related papers  (2022-06-30T21:57:07Z)
• Parameter-Efficient Sparsity for Large Language Models Fine-Tuning [63.321205487234074]
  We propose a. sparse-efficient Sparse Training (PST) method to reduce the number of trainable parameters during sparse-aware training. Experiments with diverse networks (i.e., BERT, RoBERTa and GPT-2) demonstrate PST performs on par or better than previous sparsity methods.
  arXiv  Detail & Related papers  (2022-05-23T02:43:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.