Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets

• URL: http://arxiv.org/abs/2603.16731v1
• Date: Tue, 17 Mar 2026 16:14:36 GMT
• Title: Understanding Quantization of Optimizer States in LLM Pre-training: Dynamics of State Staleness and Effectiveness of State Resets
• Abstract summary: We study low-precision exponential moving average (EMA) states and show how quantization can cause many nominal updates to round back to the same stored value.<n>We develop a simple predictive model of stalling that estimates one-step stalling probabilities and characterizes how stalling builds up over time.<n>This perspective provides a mechanistic explanation for why-state resets help in low precision.
• Score: 10.325245543844245
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quantizing optimizer states is becoming an important ingredient of memory-efficient large-scale pre-training, but the resulting optimizer dynamics remain only partially understood. We study low-precision exponential moving average (EMA) optimizer states and show how quantization can cause many nominal updates to round back to the same stored value, making the state effectively stale and slowing adaptation beyond what the nominal decay would suggest. We then develop a simple predictive model of stalling that estimates one-step stalling probabilities and characterizes how stalling builds up over time after the initialization. This perspective provides a mechanistic explanation for why optimizer-state resets help in low precision: once a quantized EMA becomes effectively stale, resetting it can temporarily restore responsiveness. Motivated by this picture, we derive a simple theory-guided method for choosing useful reset periods, showing that in low precision the key question is not only whether resets help, but when they should be applied. Experiments in controlled simulations and LLM pre-training show that suitable reset schedules recover the performance lost to low-precision state storage while substantially reducing optimizer-state memory.

Related papers

• When Quantization Breaks Memory: Recurrent-State Write-Back in Low-Precision Temporal Inference [0.0]
  In recurrent networks, the quantized state is stored and returned at the next time step, so the rule used to store that state can alter subsequent computations.<n>We introduce recurrent-state write-back to denote this rule and isolate its effect in a compact GRU encoder--decoder for fluorescence lifetime imaging.<n>Our results establish recurrent-state write-back as a key determinant of low-precision recurrent dynamics and identify the state-storage interface as a central design consideration for quantized recurrent inference.
  arXiv  Detail & Related papers  (2026-09-03T21:23:35Z)
• Stochastic Resetting Accelerates Policy Convergence in Reinforcement Learning [5.900714266080361]
  resetting is a powerful mechanism for optimizing first-passage properties.<n>We show that resetting accelerates convergence even when it does not reduce the search time of a purely diffusive agent.<n>In a continuous control task with neural-network-based value approximation, we show that random resetting improves deep reinforcement learning when exploration is difficult.
  arXiv  Detail & Related papers  (2026-03-17T17:50:32Z)
• Adaptive Visual Autoregressive Acceleration via Dual-Linkage Entropy Analysis [50.48301331112126]
  We propose NOVA, a training-free token reduction acceleration framework for Visual AutoRegressive modeling.<n>NOVA adaptively determines the acceleration activation scale during inference by online identifying the inflection point of scale entropy growth.<n>Experiments and analyses validate NOVA as a simple yet effective training-free acceleration framework.
  arXiv  Detail & Related papers  (2026-02-01T17:29:42Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• Limitations for adaptive quantum state tomography in the presence of detector noise [0.0]
  Assumption-free reconstruction of quantum states from measurements is essential for benchmarking and certifying quantum devices.<n>An approach to alleviating these demands is provided by adaptive measurement strategies.<n>We show that any nonzero readout noise eliminates the quadratic scaling advantage of adaptive strategies.
  arXiv  Detail & Related papers  (2026-01-07T15:35:29Z)
• The Curious Case of In-Training Compression of State Space Models [49.819321766705514]
  State Space Models (SSMs) tackle long sequence modeling tasks efficiently, offer both parallelizable training and fast inference.<n>Key design challenge is striking the right balance between maximizing expressivity and limiting this computational burden.<n>Our approach, textscCompreSSM, applies to Linear Time-Invariant SSMs such as Linear Recurrent Units, but is also extendable to selective models.
  arXiv  Detail & Related papers  (2025-10-03T09:02:33Z)
• End-to-End On-Device Quantization-Aware Training for LLMs at Inference Cost [53.25965863436039]
  Quantization-aware training (QAT) provides a more principled solution, but its reliance on backpropagation incurs prohibitive memory costs.<n>We propose ZeroQAT, a zeroth-order optimization-based QAT framework that supports both weight and activation quantization.<n>Experiments show that ZeroQAT consistently outperforms representative PTQ and QAT baselines while requiring significantly less memory.
  arXiv  Detail & Related papers  (2025-08-21T01:18:27Z)
• Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics [64.62231094774211]
  Statefuls (e.g., Adam) maintain auxiliary information even 2x the model size in order to achieve optimal convergence.<n>SOLO enables Adam-styles to maintain quantized states with precision as low as 3 bits, or even 2 bits.<n>SOLO can thus be seamlessly applied to Adam-styles, leading to substantial memory savings with minimal accuracy loss.
  arXiv  Detail & Related papers  (2025-05-01T06:47:45Z)
• Fast Training of Recurrent Neural Networks with Stationary State Feedbacks [48.22082789438538]
  Recurrent neural networks (RNNs) have recently demonstrated strong performance and faster inference than Transformers.<n>We propose a novel method that replaces BPTT with a fixed gradient feedback mechanism.
  arXiv  Detail & Related papers  (2025-03-29T14:45:52Z)
• Instance-dependent Early Stopping [57.912273923450726]
  We propose an Instance-dependent Early Stopping (IES) method that adapts the early stopping mechanism from the entire training set to the instance level.<n>IES considers an instance as mastered if the second-order differences of its loss value remain within a small range around zero.<n>IES can reduce backpropagation instances by 10%-50% while maintaining or even slightly improving the test accuracy and transfer learning performance of a model.
  arXiv  Detail & Related papers  (2025-02-11T13:34:09Z)
• Initial-state-dependent quantum speed limit for dissipative state preparation: Framework and optimization [6.211723927647019]
  We focus on a Markovian dissipative state preparation scheme where the prepared state is one of the energy eigenstates. We derive an initial-state-dependent quantum speed limit (QSL) that offers a more refined measure of the actual evolution time. We demonstrate the effectiveness of our strategy in a dissipative Rydberg atom system for preparing the Bell state.
  arXiv  Detail & Related papers  (2023-03-23T00:19:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.