SandwichQuant: Which Parameters Matter Before and After Quantization?

• URL: http://arxiv.org/abs/2608.24173v1
• Date: Tue, 25 Aug 2026 07:40:06 GMT
• Title: SandwichQuant: Which Parameters Matter Before and After Quantization?
• Abstract summary: We study quantization correction from a parameter subspace perspective.<n>We show that the low-dimensional normalization-affine subspace provides a highly efficient correction direction under matched budgets.<n>We propose SandwichQuant, a two-stage normalization-affine correction framework.
• Score: 8.132552404141467
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Quantization correction methods usually optimize weights, quantization parameters, or reconstruction objectives, while the underlying parameter subspaces responsible for effective correction remain unclear. In this work, we study quantization correction from a parameter subspace perspective and reveal that correction capability is highly non-uniform across parameter groups. By decomposing trainable parameters into backbone weights, normalization-affine parameters, and quantization parameters, we show that the low-dimensional normalization-affine subspace provides a highly efficient correction direction under matched budgets. Based on this finding, we propose SandwichQuant, a two-stage normalization-affine correction framework that performs adaptation before and after quantization. The pre-stage improves quantization robustness, while the post-stage compensates residual errors after the quantized graph is fixed. Extensive experiments on vision models and large language models demonstrate consistent improvements under various low-bit quantization settings, validating the effectiveness of subspace-aligned correction.

Related papers

• Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization [12.719434226050522]
  Low-bit quantization suffers severe accuracy degradation on compact networks.<n> Conventional pipelines adopt monolithic optimization.<n>We propose Normalization Affine Preconditioning (NAP) for targeted subspace optimization.
  arXiv  Detail & Related papers  (2026-08-04T16:50:04Z)
• Saliency-Aware Regularized Quantization Calibration for Large Language Models [12.382772717031772]
  Post-training quantization (PTQ) is an effective approach for deploying large language models (LLMs) under memory and latency constraints.<n>Most existing PTQ methods determine quantization parameters by minimizing a layer-wise reconstruction error on a predetermined calibration dataset.<n>The proposed regularization encourages quantized weights to remain close to the original weights during calibration, leading to improved generalization at inference time.
  arXiv  Detail & Related papers  (2026-05-07T05:33:26Z)
• Regularized Calibration with Successive Rounding for Post-Training Quantization [32.31386646428613]
  Post-training quantization (PTQ) enables efficient inference by mapping pretrained weights to low-bit formats without retraining.<n>We show that interpolating between symmetric and asymmetric calibration acts as a form of regularization.<n>We derive a simple successive rounding procedure that naturally incorporates asymmetric calibration.
  arXiv  Detail & Related papers  (2026-02-05T17:18:02Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• Beyond Outliers: A Study of Optimizers Under Quantization [82.75879062804955]
  We study impact of choice on model robustness under quantization.<n>We evaluate how model performance degrades when trained with different baselines.<n>We derive scaling laws for quantization-aware training under different parameters.
  arXiv  Detail & Related papers  (2025-09-27T21:15:22Z)
• Energy Landscape Plummeting in Variational Quantum Eigensolver: Subspace Optimization, Non-iterative Corrections and Generator-informed Initialization for Improved Quantum Efficiency [0.0]
  Variational Quantum Eigensolver (VQE) faces significant challenges due to hardware noise and the presence of barren plateaus and local traps.<n>We introduce a general formalism that optimize hardware resource utilization and accuracy by projecting VQE optimizations on to a reduced-dimensional subspace.<n> Numerical simulations show that, when integrated with any chemistry-inspired ansatz, our method can provide one to two orders of magnitude better estimation of the minima.
  arXiv  Detail & Related papers  (2025-04-17T17:07:09Z)
• Scaling Exponents Across Parameterizations and Optimizers [94.54718325264218]
  We propose a new perspective on parameterization by investigating a key assumption in prior work. Our empirical investigation includes tens of thousands of models trained with all combinations of threes. We find that the best learning rate scaling prescription would often have been excluded by the assumptions in prior work.
  arXiv  Detail & Related papers  (2024-07-08T12:32:51Z)
• Maintaining Structural Integrity in Parameter Spaces for Parameter Efficient Fine-tuning [78.39310274926535]
  Adapting pre-trained foundation models for various downstream tasks has been prevalent in artificial intelligence.<n>To mitigate this, several fine-tuning techniques have been developed to update the pre-trained model weights in a more resource-efficient manner.<n>This paper introduces a generalized parameter-efficient fine-tuning framework, designed for various dimensional parameter space.
  arXiv  Detail & Related papers  (2024-05-23T16:04:42Z)
• Towards Accurate Post-training Quantization for Reparameterized Models [6.158896686945439]
  Current Post-training Quantization (PTQ) methods often lead to significant accuracy degradation. This is primarily caused by channel-specific and sample-specific outliers. We propose RepAPQ, a novel framework that preserves the accuracy of quantized reparameterization models.
  arXiv  Detail & Related papers  (2024-02-25T15:42:12Z)
• PreQuant: A Task-agnostic Quantization Approach for Pre-trained Language Models [52.09865918265002]
  We propose a novel quantize before fine-tuning'' framework, PreQuant. PreQuant is compatible with various quantization strategies, with outlier-aware fine-tuning incorporated to correct the induced quantization error. We demonstrate the effectiveness of PreQuant on the GLUE benchmark using BERT, RoBERTa, and T5.
  arXiv  Detail & Related papers  (2023-05-30T08:41:33Z)
• Spectral Tensor Train Parameterization of Deep Learning Layers [136.4761580842396]
  We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. We show the effects of neural network compression in the classification setting and both compression and improved stability training in the generative adversarial training setting.
  arXiv  Detail & Related papers  (2021-03-07T00:15:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.