Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics

• URL: http://arxiv.org/abs/2606.19367v1
• Date: Thu, 11 Jun 2026 03:50:14 GMT
• Title: Weibull Weight-Scale Parameter Evolution under AdamW Training Dynamics
• Abstract summary: We study why the Weibull weight-scale parameter $(t)$ grows, overshoots, and then relaxes during AdamW training.<n>Near saturation, alignment and decay approach balance, explaining the transition from weight-scale growth to relaxation.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Building on a two-parameter Weibull framework for diagnosing transformer weight distributions, we study why the Weibull weight-scale parameter $λ$ grows, overshoots, and then relaxes during AdamW training. We derive a leading-order three-force decomposition of the squared weight norm from the AdamW update: an alignment force measuring the correlation between weights and the adaptive update direction, an injection force from adaptive step magnitude, and a decay force from decoupled weight decay. On self-trained Pythia-70M models with ground-truth optimizer moments, alignment dominates the rise phase, contributing 88-94% of the absolute force budget across four random seeds and remaining robust to super-weight removal. Near saturation, alignment and decay approach balance, explaining the transition from weight-scale growth to relaxation. These force dynamics directly govern the squared-norm component underlying $λ(t)$; the remaining RMS-to-Weibull reconstruction offset is measurable and decomposes into bridge and integration components, totaling approximately 5-6% in densely sampled regions. To extend the analysis to real models where optimizer moments are unavailable, we introduce a spline displacement method that recovers the alignment force from sparse checkpoints with approximately 92-94% accuracy, about twice the naive two-point baseline. We further observe that the peak value of $λ(t)$ varies with training-data coherence in our experiments, suggesting a data-dependent component of weight-scale growth that we leave to a controlled follow-up study. Code and data are available at https://github.com/tiexinding/NPM-Weibull-public.

Related papers

• Improving Neural Network Training by Decoupling the Magnitude and Direction of Weight Vectors [110.10775872372047]
  We propose emphMagnitude-Direction-Direction (MD) Decoupling.<n>It factorizes each weight into a fixed-norm direction on a hypersphere and learnable per-row and per-column magnitude gains, updated at separate learning rates.<n>Across both Adam and Muon, MD Decoupling improves on well-tuned baselines, transfers the optimal LR across model width without retuning, and continues to help at scale on large Mixture-of-Experts models.
  arXiv  Detail & Related papers  (2026-06-24T15:40:26Z)
• Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization [67.29294986811314]
  Matrix baseds such as Muon can substantially speed up language model pretraining, but their gains over AdamW are observed to shrink when using standard constant weight decay.<n>We propose Hyperball, a simple wrapper that addresses this issue.<n>On Qwen3 style models up to 1.2B parameters, Muon Hyperball achieves 20-30% token equivalent speedup over weight decay baselines.
  arXiv  Detail & Related papers  (2026-06-15T16:09:02Z)
• How Neural Reward Models Learn Features for Policy Optimization: A Single-Index Analysis [53.063298916923976]
  We study the feedback in a Gaussian single-index model with $r*(x) = *(langle *, xrangle)$ and $x sim N(0, I_d)$.<n>We analyze a two-stage neural reward model that first learns the hidden direction $*$ from reward-weighted samples and then fits the readout layer by weighted ridge regression.
  arXiv  Detail & Related papers  (2026-05-23T22:00:38Z)
• Normalized Architectures are Natively 4-Bit [49.13186675123547]
  nGPT, an architecture that constrains weights and hidden representations to the unit hypersphere, is inherently more robust to low-precision arithmetic.<n>We validate this approach on both a 1.2B dense model and hybrid (Mamba-Transformer) MoE models of up to 3B/30B parameters.
  arXiv  Detail & Related papers  (2026-05-07T11:54:07Z)
• Decoupled Orthogonal Dynamics: Regularization for Deep Network Optimizers [7.170491058218013]
  We argue that magnitude and direction play distinct roles and should be decoupled in dynamics.<n>We propose Orthogonal Dynamics Decoupling and instantiate it as AdamO: an SGD-style update handles the onedimensional norm control, while Adam's adaptive preconditioning is confined to the tangential subspace.<n> Experiments on vision and language tasks show that AdamO improves generalization and stability over AdamW without introducing additional complex constraints.
  arXiv  Detail & Related papers  (2026-02-04T23:41:51Z)
• ECO: Quantized Training without Full-Precision Master Weights [58.97082407934466]
  Error-Compensating (ECO) eliminates master weights by applying updates directly to quantized parameters.<n>We show that ECO converges to a constant-radius neighborhood of the optimum, while naive master-weight removal can incur an error that is inversely proportional to the learning rate.
  arXiv  Detail & Related papers  (2026-01-29T18:35:01Z)
• Correction of Decoupled Weight Decay [2.538209532048867]
  We argue that decoupled weight decay $propto 2$ leads to stable weight norm based on simple assumption that updates become independent of steady state.<n>We show that decoupled weight decay $propto 2$ leads to stable weight and gradient norms and allows us to better control the training dynamics and improve the model performance.
  arXiv  Detail & Related papers  (2025-12-09T03:52:54Z)
• AdamHD: Decoupled Huber Decay Regularization for Language Model Pre-Training [0.2578242050187029]
  AdamHuberDecay is a drop-in replacement for AdamW that substitutes the $ell$ penalty with a decoupled smooth Huber regularizer.<n>Experiments on GPT-2 and GPT-3 pre-training demonstrate that AdamHuberDecay converges 10-15% faster in wall-clock time.
  arXiv  Detail & Related papers  (2025-11-18T18:08:20Z)
• Robust Layerwise Scaling Rules by Proper Weight Decay Tuning [50.11170157029911]
  In modern scale-invariant architectures, training quickly enters an degrading-governed steady state.<n>We introduce a weight-decay scaling rule for AdamW that preserves sublayer gain across widths.<n>Our results extend $mu$P beyond the near-init regime by explicitly controlling the steady-state scales set by parameters.
  arXiv  Detail & Related papers  (2025-10-17T02:58:35Z)
• Generalized Balancing Weights via Deep Neural Networks [0.0]
  Estimating causal effects from observational data is a central problem in many domains. We present generalized balancing weights, Neural Balancing Weights (NBW), to estimate the causal effects of an arbitrary mixture of discrete and continuous interventions.
  arXiv  Detail & Related papers  (2022-11-14T17:03:56Z)
• Learning to Re-weight Examples with Optimal Transport for Imbalanced Classification [74.62203971625173]
  Imbalanced data pose challenges for deep learning based classification models. One of the most widely-used approaches for tackling imbalanced data is re-weighting. We propose a novel re-weighting method based on optimal transport (OT) from a distributional point of view.
  arXiv  Detail & Related papers  (2022-08-05T01:23:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.