Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization

• URL: http://arxiv.org/abs/2608.03919v1
• Date: Tue, 04 Aug 2026 16:50:04 GMT
• Title: Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization
• Abstract summary: Low-bit quantization suffers severe accuracy degradation on compact networks.<n> Conventional pipelines adopt monolithic optimization.<n>We propose Normalization Affine Preconditioning (NAP) for targeted subspace optimization.
• Score: 12.719434226050522
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Low-bit quantization suffers severe accuracy degradation on compact networks, rooted in the dominant full-parameter coupled training paradigm that ignores parameter subspace heterogeneity. Their limited feature redundancy leaves little room to absorb quantization errors. Conventional pipelines adopt monolithic optimization: PTQ reconstructs fixed pretrained models without improving inherent quantization friendliness; QAT updates all parameters jointly, suffering from gradient coupling between backbone weights and calibration parameters. In this paper, we identify normalization affine parameters as a low-dimensional high-leverage subspace dominating quantization robustness, and propose Normalization Affine Preconditioning (NAP) for targeted subspace optimization. For PTQ, NAP freezes backbone weights and fine-tunes only affine parameters under the target fake-quantization graph on full-precision models, proactively boosting quantization friendliness before downstream reconstruction. For QAT, we introduce an alternating QAT-NAP schema that decouples feature learning and numerical calibration, breaking the performance ceiling of saturated joint training. Theoretical analysis confirms BN affine parameters fully cancel the channel-wise affine component of quantization distortion, while nonlinear rounding and clipping residuals form the irreducible error boundary; distillation-guided NAP acts as directional flatness optimization, projecting teacher-student logit mismatch onto the restricted subspace. Experiments on ImageNet and CIFAR-100 show NAP recovers severely collapsed low-bit quantization, consistently boosts reconstruction-based PTQ, and outperforms saturated full-parameter QAT with negligible tuning cost. This work reveals the principle of targeted low-dimensional subspace optimization, offering a new perspective beyond full-parameter coupled training for efficient deep learning.

Related papers

• SandwichQuant: Which Parameters Matter Before and After Quantization? [8.132552404141467]
  We study quantization correction from a parameter subspace perspective.<n>We show that the low-dimensional normalization-affine subspace provides a highly efficient correction direction under matched budgets.<n>We propose SandwichQuant, a two-stage normalization-affine correction framework.
  arXiv  Detail & Related papers  (2026-08-25T07:40:06Z)
• Neural Network Quantization by Learning Low-Loss Subspaces [42.07461306663832]
  Recent studies indicate that low-loss FP solutions are not isolated, but instead belong to connected low-loss subspaces of the loss landscape.<n>This raises the question: can a quantized model be constructed to lie within a low-loss subspace of the FP model, thereby automatically preserving performance?<n>We address this question by learning quantization-aware linear paths in weight space optimized to minimize loss.<n>We demonstrate that the midpoint of the resulting subspace is, by design, quantization-friendly and that its direct quantization yields performance comparable to that of quantization-aware training.
  arXiv  Detail & Related papers  (2026-06-23T18:48:45Z)
• PEFT-Arena: Understanding Parameter-Efficient Finetuning from a Stability-Plasticity Perspective [52.693471818837395]
  We argue that PEFT should be assessed through the stability-plasticity dilemma.<n>We introduce PEFT-Arena, a benchmark that measures downstream performance and general capability retention.<n>Inspired by this, we present case studies of a post-hoc improvement with path-wise rewinding.
  arXiv  Detail & Related papers  (2026-05-27T17:59:51Z)
• Quantum Tilted Loss in Variational Optimization: Theory and Applications [2.274915755738124]
  We introduce the Quantum Tilted Loss (QTL) to reshape the optimization landscape.<n>QTL can amplify gradient signals in structured settings while preserving the problem's true global minima.<n>We formalize this trainability-estimability trade-off, demonstrating how aggressive tilting fundamentally shifts the optimization bottleneck.
  arXiv  Detail & Related papers  (2026-05-04T17:25:43Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• LOTION: Smoothing the Optimization Landscape for Quantized Training [9.855789447735757]
  We introduce LOTION, a principled smoothing framework that replaces the raw quantized loss with its expectation under unbiased randomized-rounding noise.<n>We empirically demonstrate that this method outperforms standard QAT on synthetic testbeds and on 150M- and 300M- parameter language models.
  arXiv  Detail & Related papers  (2025-10-09T19:16:46Z)
• Beyond Outliers: A Study of Optimizers Under Quantization [82.75879062804955]
  We study impact of choice on model robustness under quantization.<n>We evaluate how model performance degrades when trained with different baselines.<n>We derive scaling laws for quantization-aware training under different parameters.
  arXiv  Detail & Related papers  (2025-09-27T21:15:22Z)
• Progressive Element-wise Gradient Estimation for Neural Network Quantization [2.1413624861650358]
  Quantization-Aware Training (QAT) methods rely on the Straight-Through Estimator (STE) to address the non-differentiability of discretization functions.<n>We propose Progressive Element-wise Gradient Estimation (PEGE) to address discretization errors between continuous and quantized values.<n>PEGE consistently outperforms existing backpropagation methods and enables low-precision models to match or even outperform the accuracy of their full-precision counterparts.
  arXiv  Detail & Related papers  (2025-08-27T15:59:36Z)
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost [53.25965863436039]
  Quantization-aware training (QAT) provides a more principled solution, but its reliance on backpropagation incurs prohibitive memory costs.<n>We propose ZeroQAT, a zeroth-order optimization-based QAT framework that supports both weight and activation quantization.<n>Experiments show that ZeroQAT consistently outperforms representative PTQ and QAT baselines while requiring significantly less memory.
  arXiv  Detail & Related papers  (2025-08-21T01:18:27Z)
• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation [55.12070409045766]
  Post-training quantization (PTQ) has stood out as a cost-effective and promising model compression paradigm in recent years.<n>Current PTQ methods for Vision Transformers (ViTs) still suffer from significant accuracy degradation, especially under low-bit quantization.
  arXiv  Detail & Related papers  (2025-06-13T07:57:38Z)
• NIPQ: Noise proxy-based Integrated Pseudo-Quantization [9.207644534257543]
  Straight-through estimator (STE) incurs unstable convergence during quantization-aware training (QAT) We propose a novel noise proxy-based integrated pseudoquantization (NIPQ) that enables unified support of pseudoquantization for both activation and weight. NIPQ outperforms existing quantization algorithms in various vision and language applications by a large margin.
  arXiv  Detail & Related papers  (2022-06-02T01:17:40Z)
• Spectral Tensor Train Parameterization of Deep Learning Layers [136.4761580842396]
  We study low-rank parameterizations of weight matrices with embedded spectral properties in the Deep Learning context. We show the effects of neural network compression in the classification setting and both compression and improved stability training in the generative adversarial training setting.
  arXiv  Detail & Related papers  (2021-03-07T00:15:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.