Neural Network Quantization by Learning Low-Loss Subspaces

• URL: http://arxiv.org/abs/2606.25087v1
• Date: Tue, 23 Jun 2026 18:48:45 GMT
• Title: Neural Network Quantization by Learning Low-Loss Subspaces
• Abstract summary: Recent studies indicate that low-loss FP solutions are not isolated, but instead belong to connected low-loss subspaces of the loss landscape.<n>This raises the question: can a quantized model be constructed to lie within a low-loss subspace of the FP model, thereby automatically preserving performance?<n>We address this question by learning quantization-aware linear paths in weight space optimized to minimize loss.<n>We demonstrate that the midpoint of the resulting subspace is, by design, quantization-friendly and that its direct quantization yields performance comparable to that of quantization-aware training.
• Score: 42.07461306663832
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Neural network quantization aims to find a discrete representation of parameters that preserves the performance of a full-precision (FP) model as faithfully as possible. Enforcing discrete constraints perturbs parameters away from a well-optimized minimum, generally resulting in performance degradation. Recent studies indicate that low-loss FP solutions are not isolated, but instead belong to connected low-loss subspaces of the loss landscape, where the loss maintains nearly the same minimum value. Models sampled from these subspaces are diverse and retain high accuracy. This raises the question: can a quantized model be constructed to lie within a low-loss subspace of the FP model, thereby automatically preserving performance? We address this question by learning quantization-aware linear paths in weight space optimized to minimize loss. We demonstrate that the midpoint of the resulting subspace is, by design, quantization-friendly and that its direct quantization yields performance comparable to that of quantization-aware training. The proposed procedure offers a novel perspective on weight quantization and, in contrast to conventional methods, neither relies on the straight-through estimator nor involves explicit discretization during training.

Related papers

• Scalable Lindblad Noise Learning via Stochastic Tensor-Network Simulation [48.887627688666974]
  Learning dissipation rates in large-scale open quantum systems is a major obstacle for near-term quantum technologies.<n>Here, we propose a scalable noise-learning framework for Lindblad dissipation rates that combines a simulation method, the Jump Method (TJM)<n>We demonstrate the approach on two noise models in the Ising model, in which independent dissipation rates are learned for each site up to $N_mathrmsite=16$.<n>This combination of scalable numerics and rigorous theoretical guarantees positions TJM-based noise learning as a practical foundation for characterizing dissipation in large quantum devices.
  arXiv  Detail & Related papers  (2026-08-25T15:07:11Z)
• Low-Dimensional High-Leverage Subspace Optimization: Beyond Full-Parameter Coupled Training for Neural Network Quantization [12.719434226050522]
  Low-bit quantization suffers severe accuracy degradation on compact networks.<n> Conventional pipelines adopt monolithic optimization.<n>We propose Normalization Affine Preconditioning (NAP) for targeted subspace optimization.
  arXiv  Detail & Related papers  (2026-08-04T16:50:04Z)
• Enhanced Neural Quantum State via Annealed Gradient Descent [12.313934054741928]
  Annealed gradient descent suppresses metastable trapping, preserves physically relevant configurations and enables compact neural quantum states to attain chemical accuracy and competitive state-of-the-art performance.<n>These results establish AGD as a lightweight, complement to expressive neural architectures, improved for scalable quantum many-body optimization.
  arXiv  Detail & Related papers  (2026-07-21T08:58:10Z)
• Fitting Is Not Enough: Smoothness in Extremely Quantized LLMs [46.88109820641649]
  Large language models (LLMs) achieve strong performance but incur high deployment costs.<n>Existing quantization algorithms mainly focus on improving the numerical accuracy of forward computation.<n>We show that extremely quantized LLMs suffer from systematic degradation beyond numerical precision loss.
  arXiv  Detail & Related papers  (2026-05-09T11:19:51Z)
• Low-Precision Streaming PCA [19.00482065580274]
  Low-precision streaming PCA estimates the top principal component in a streaming setting under limited precision.<n>We study Oja's algorithm for streaming PCA under linear and nonlinear quantization.
  arXiv  Detail & Related papers  (2025-10-25T21:48:17Z)
• LOTION: Smoothing the Optimization Landscape for Quantized Training [9.855789447735757]
  We introduce LOTION, a principled smoothing framework that replaces the raw quantized loss with its expectation under unbiased randomized-rounding noise.<n>We empirically demonstrate that this method outperforms standard QAT on synthetic testbeds and on 150M- and 300M- parameter language models.
  arXiv  Detail & Related papers  (2025-10-09T19:16:46Z)
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost [53.25965863436039]
  Quantization-aware training (QAT) provides a more principled solution, but its reliance on backpropagation incurs prohibitive memory costs.<n>We propose ZeroQAT, a zeroth-order optimization-based QAT framework that supports both weight and activation quantization.<n>Experiments show that ZeroQAT consistently outperforms representative PTQ and QAT baselines while requiring significantly less memory.
  arXiv  Detail & Related papers  (2025-08-21T01:18:27Z)
• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• Boost Post-Training Quantization via Null Space Optimization for Large Language Models [66.73751310500656]
  Existing post-training quantization methods for large language models (LLMs) offer remarkable success.<n>The increasingly marginal performance gains suggest that existing quantization strategies are insufficient to support the development of more compressed models.<n>We argue that the quantization error can be effectively alleviated by constraining the post-quantization weight to lie within the null space of input activations.
  arXiv  Detail & Related papers  (2025-05-21T14:07:07Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE)<n>RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers.<n>Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• Understanding the Difficulty of Low-Precision Post-Training Quantization for LLMs [4.5529796609245805]
  Large language models of high parameter counts are computationally expensive, yet can be made much more efficient by compressing their weights to very low numerical precision.<n>We show that, under the same data constraint, the former approach nearly always fared worse than the latter, a phenomenon particularly prominent when the numerical precision is very low.
  arXiv  Detail & Related papers  (2024-10-18T16:16:52Z)
• DAQ: Distribution-Aware Quantization for Deep Image Super-Resolution Networks [49.191062785007006]
  Quantizing deep convolutional neural networks for image super-resolution substantially reduces their computational costs. Existing works either suffer from a severe performance drop in ultra-low precision of 4 or lower bit-widths, or require a heavy fine-tuning process to recover the performance. We propose a novel distribution-aware quantization scheme (DAQ) which facilitates accurate training-free quantization in ultra-low precision.
  arXiv  Detail & Related papers  (2020-12-21T10:19:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.