Low-Precision Streaming PCA

• URL: http://arxiv.org/abs/2510.22440v1
• Date: Sat, 25 Oct 2025 21:48:17 GMT
• Title: Low-Precision Streaming PCA
• Abstract summary: Low-precision streaming PCA estimates the top principal component in a streaming setting under limited precision.<n>We study Oja's algorithm for streaming PCA under linear and nonlinear quantization.
• Score: 19.00482065580274
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-precision streaming PCA estimates the top principal component in a streaming setting under limited precision. We establish an information-theoretic lower bound on the quantization resolution required to achieve a target accuracy for the leading eigenvector. We study Oja's algorithm for streaming PCA under linear and nonlinear stochastic quantization. The quantized variants use unbiased stochastic quantization of the weight vector and the updates. Under mild moment and spectral-gap assumptions on the data distribution, we show that a batched version achieves the lower bound up to logarithmic factors under both schemes. This leads to a nearly dimension-free quantization error in the nonlinear quantization setting. Empirical evaluations on synthetic streams validate our theoretical findings and demonstrate that our low-precision methods closely track the performance of standard Oja's algorithm.

Related papers

• Neural Network Quantization by Learning Low-Loss Subspaces [42.07461306663832]
  Recent studies indicate that low-loss FP solutions are not isolated, but instead belong to connected low-loss subspaces of the loss landscape.<n>This raises the question: can a quantized model be constructed to lie within a low-loss subspace of the FP model, thereby automatically preserving performance?<n>We address this question by learning quantization-aware linear paths in weight space optimized to minimize loss.<n>We demonstrate that the midpoint of the resulting subspace is, by design, quantization-friendly and that its direct quantization yields performance comparable to that of quantization-aware training.
  arXiv  Detail & Related papers  (2026-06-23T18:48:45Z)
• IQP Born Machines under Data-dependent and Agnostic Initialization Strategies [0.0]
  Quantum circuit Born machines based on instantaneous quantum-time (IQP) circuits are natural candidates for quantum generative modeling.<n>Recent proposals focus on training IQP-QCBMs using Mean Maximum Discrepancy (MMD) losses built from low-body Pauli-$Z$ors.
  arXiv  Detail & Related papers  (2026-03-15T19:54:04Z)
• CAGE: Curvature-Aware Gradient Estimation For Accurate Quantization-Aware Training [73.46600457802693]
  We introduce a new method that counteracts the loss induced by quantization.<n>CAGE significantly improves upon the state-of-theart methods in terms of accuracy, for similar computational cost.<n>For QAT pre-training of Llama models, CAGE matches the accuracy achieved at 4-bits (W4A4) with the prior best method.
  arXiv  Detail & Related papers  (2025-10-21T16:33:57Z)
• Provable Post-Training Quantization: Theoretical Analysis of OPTQ and Qronos [11.469337174377046]
  Post-training quantization (PTQ) has become a crucial tool for reducing the memory and compute costs of modern deep neural networks.<n>OPTQ framework-also known as GPTQ-has emerged as a leading method due to its computational efficiency and strong empirical performance.<n>Despite its widespread adoption, OPTQ lacks rigorous quantitative theoretical guarantees.
  arXiv  Detail & Related papers  (2025-08-06T20:00:40Z)
• MPQ-DMv2: Flexible Residual Mixed Precision Quantization for Low-Bit Diffusion Models with Temporal Distillation [74.34220141721231]
  We present MPQ-DMv2, an improved textbfMixed textbfPrecision textbfQuantization framework for extremely low-bit textbfDiffusion textbfModels.
  arXiv  Detail & Related papers  (2025-07-06T08:16:50Z)
• Beyond Sin-Squared Error: Linear-Time Entrywise Uncertainty Quantification for Streaming PCA [5.749787074942513]
  We propose a novel statistical inference framework for streaming principal component analysis (PCA) using Oja's algorithm.<n>We derive a sharp Bernstein-type concentration bound for elements of the estimated vector matching the optimal error rate up to logarithmic factors.<n>To efficiently estimate the coordinate-wise variance, we introduce a provably consistent subsampling algorithm.
  arXiv  Detail & Related papers  (2025-06-14T22:50:54Z)
• FIMA-Q: Post-Training Quantization for Vision Transformers by Fisher Information Matrix Approximation [55.12070409045766]
  Post-training quantization (PTQ) has stood out as a cost-effective and promising model compression paradigm in recent years.<n>Current PTQ methods for Vision Transformers (ViTs) still suffer from significant accuracy degradation, especially under low-bit quantization.
  arXiv  Detail & Related papers  (2025-06-13T07:57:38Z)
• RoSTE: An Efficient Quantization-Aware Supervised Fine-Tuning Approach for Large Language Models [53.571195477043496]
  We propose an algorithm named Rotated Straight-Through-Estimator (RoSTE)<n>RoSTE combines quantization-aware supervised fine-tuning (QA-SFT) with an adaptive rotation strategy to reduce activation outliers.<n>Our findings reveal that the prediction error is directly proportional to the quantization error of the converged weights, which can be effectively managed through an optimized rotation configuration.
  arXiv  Detail & Related papers  (2025-02-13T06:44:33Z)
• OAC: Output-adaptive Calibration for Accurate Post-training Quantization [28.67781845829386]
  Post-training Quantization (PTQ) techniques have been developed to compress Large Language Models (LLMs)<n>Most PTQ approaches formulate the quantization error based on a layer-wise Euclidean loss, ignoring the model output.<n>We propose Output-adaptive Quantization (OAC) to incorporate the model output in the calibration process.
  arXiv  Detail & Related papers  (2024-05-23T20:01:17Z)
• Regularized Vector Quantization for Tokenized Image Synthesis [126.96880843754066]
  Quantizing images into discrete representations has been a fundamental problem in unified generative modeling. deterministic quantization suffers from severe codebook collapse and misalignment with inference stage while quantization suffers from low codebook utilization and reconstruction objective. This paper presents a regularized vector quantization framework that allows to mitigate perturbed above issues effectively by applying regularization from two perspectives.
  arXiv  Detail & Related papers  (2023-03-11T15:20:54Z)
• An adjoint-free algorithm for conditional nonlinear optimal perturbations (CNOPs) via sampling [5.758073912084367]
  We propose a sampling algorithm based on state-of-the-art statistical machine learning techniques to obtain conditional nonlinear optimal perturbations (CNOPs) The sampling approach directly reduces the gradient to the objective function value (zeroth-order information) We demonstrate the CNOPs obtained with their spatial patterns, objective values, quantifying computation times, and nonlinear error growth.
  arXiv  Detail & Related papers  (2022-08-01T16:07:22Z)
• Deep Equilibrium Optical Flow Estimation [80.80992684796566]
  Recent state-of-the-art (SOTA) optical flow models use finite-step recurrent update operations to emulate traditional algorithms. These RNNs impose large computation and memory overheads, and are not directly trained to model such stable estimation. We propose deep equilibrium (DEQ) flow estimators, an approach that directly solves for the flow as the infinite-level fixed point of an implicit layer.
  arXiv  Detail & Related papers  (2022-04-18T17:53:44Z)
• Q-Rater: Non-Convex Optimization for Post-Training Uniform Quantization [9.062897838978955]
  Various post-training quant uniformization methods have usually been based on convex optimization. Our proposed technique presents higher model accuracy, especially for a low quantization.
  arXiv  Detail & Related papers  (2021-05-05T05:14:22Z)
• Amortized Conditional Normalized Maximum Likelihood: Reliable Out of Distribution Uncertainty Estimation [99.92568326314667]
  We propose the amortized conditional normalized maximum likelihood (ACNML) method as a scalable general-purpose approach for uncertainty estimation. Our algorithm builds on the conditional normalized maximum likelihood (CNML) coding scheme, which has minimax optimal properties according to the minimum description length principle. We demonstrate that ACNML compares favorably to a number of prior techniques for uncertainty estimation in terms of calibration on out-of-distribution inputs.
  arXiv  Detail & Related papers  (2020-11-05T08:04:34Z)
• VecQ: Minimal Loss DNN Model Compression With Vectorized Weight Quantization [19.66522714831141]
  We develop a new quantization solution called VecQ, which can guarantee minimal direct quantization loss and better model accuracy. In addition, in order to up the proposed quantization process during training, we accelerate the quantization process with a parameterized estimation and probability-based calculation.
  arXiv  Detail & Related papers  (2020-05-18T07:38:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.