Uniform Scaling Limits in AdamW-Trained Transformers

• URL: http://arxiv.org/abs/2605.11059v1
• Date: Mon, 11 May 2026 16:54:46 GMT
• Title: Uniform Scaling Limits in AdamW-Trained Transformers
• Abstract summary: We study the large-depth limit of transformers trained with AdamW.<n>We prove that the joint dynamics of the hidden states and backpropagated variables converge in $L2$.
• Score: 0.21485350418225238
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the large-depth limit of transformers trained with AdamW, by modelling the hidden-state dynamics as an interacting particle system (IPS) coupled through the attention mechanism. Under appropriate scaling of the attention heads, we prove that the joint dynamics of the hidden states and backpropagated variables converge in $L^2$, uniformly over the initial condition, to the solution of a forward--backward system of ODEs at rate $\mathcal O(L^{-1}+L^{-1/3}H^{-1/2})$. Here, $L$ and $H$ denote the depth and number of heads of the transformer, respectively. The limiting system of ODEs can be identified with a McKean--Vlasov ODE (MVODE) when the attention heads do not incorporate causal masking. By using the flow maps associated with this MVODE and applying concentration of measure techniques, we obtain bounds on the difference between the discrete and continuous models that are uniform over compact sets of initial conditions. As this is achieved without resorting to a covering argument, the constants in our bounds are independent of the number of tokens. Furthermore, under a suitable adaptation to AdamW, the bounds become independent of the token embedding dimension.

Related papers

• A Compositional Theory of Causally Masked Transformers [50.88598486616582]
  We develop a formalization that derives expressivity directly from the model's implemented dynamics.<n>Each attention head updates its own state independently within a layer.<n>A modified form of soft attention supports irreversible, checklist-like state.
  arXiv  Detail & Related papers  (2026-07-29T14:47:19Z)
• Representational Capacity: Geometric Limits on Feature Representation in Transformer Language Models [51.56484100374058]
  We develop a framework for estimating how many near-orthogonal directions a model can support.<n>Applying this metric across dozens of open-source models reveals two classes: models with high $varepsilon$ whose embeddings lack near-orthogonal structure, and models with low $varepsilon$ that maintain it.
  arXiv  Detail & Related papers  (2026-06-01T18:28:56Z)
• Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models [6.60601600774899]
  We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer with Multi-Layer Perceptron (MLP) blocks.<n>We also identify the partial differential equation describing the evolution of the tokens' distribution in this limit and prove propagation of chaos when the number of such tokens is large.
  arXiv  Detail & Related papers  (2026-04-29T17:09:05Z)
• Comparison of the standard and dressed-picture master equations for the quantum Rabi model in the ultrastrong coupling regime [78.55450146498553]
  This chapter investigates the effects of relaxation and dephasing on the quantum Rabi model in the ultrastrong coupling regime.<n>It provides explicit formulas to implement and numerically solve the resulting nonunitary dynamics.
  arXiv  Detail & Related papers  (2026-04-10T01:22:07Z)
• Homogenized Transformers [5.290251602267728]
  We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads.<n>We prove that, under suitable joint scalings of the depth, the residual step size, and the number of heads, this dynamics admits a nontrivialed limit.
  arXiv  Detail & Related papers  (2026-04-02T12:39:27Z)
• ResNets of All Shapes and Sizes: Convergence of Training Dynamics in the Large-scale Limit [11.482815450838887]
  We establish convergence of the training dynamics of residual neural networks (ResNets) to their joint infinite depth L, hidden width M, and embedding dimension D limit.<n>This work completes the program initiated in the companion paper [Chi25] where it is proved that for a fixed embedding dimension D, the training dynamics converges to a Mean ODE model.
  arXiv  Detail & Related papers  (2026-03-18T18:07:02Z)
• MambaX: Image Super-Resolution with State Predictive Control [48.76194230142064]
  Mamba has emerged as a promising approach that can represent the entire reconstruction process as a state sequence with multiple nodes, allowing for intermediate intervention.<n>We created a nonlinear state predictive control model textbfMambaX that maps consecutive spectral bands into a latent state space and generalizes the SR task by dynamically learning the nonlinear state parameters of control equations.<n>Our evaluation demonstrates the superior performance of the dynamic spectrum-state representation model in both single-image SR and multimodal fusion-based SR tasks.
  arXiv  Detail & Related papers  (2025-11-22T11:44:09Z)
• Approximation of diffeomorphisms for quantum state transfers [49.1574468325115]
  We seek to combine two emerging standpoints in control theory.<n>We numerically find control laws driving state transitions in small time in a bilinear Schr"odinger PDE posed on the torus.
  arXiv  Detail & Related papers  (2025-03-18T17:28:59Z)
• Ancilla quantum measurements on interacting chains: Sensitivity of entanglement dynamics to the type and concentration of detectors [46.76612530830571]
  We consider a quantum many-body lattice system that is coupled to ancillary degrees of freedom (detectors'') We explore the dynamics of density and of entanglement entropy in the chain, for various values of $rho_a$ and $M$.
  arXiv  Detail & Related papers  (2023-11-21T21:41:11Z)
• The Shaped Transformer: Attention Models in the Infinite Depth-and-Width Limit [38.89510345229949]
  We study the covariance matrix of a modified Softmax-based attention model with skip connections in the proportional limit of infinite-depth-and-width. To achieve a well-defined limit, the Transformer's attention mechanism is modified by centering the Softmax output at identity. We show, through simulations, that the differential equation (SDE) indexed by the depth-to-width ratio provides a surprisingly good description of the corresponding finite-size model.
  arXiv  Detail & Related papers  (2023-06-30T16:10:36Z)
• High-dimensional limit theorems for SGD: Effective dynamics and critical scaling [6.950316788263433]
  We prove limit theorems for the trajectories of summary statistics of gradient descent (SGD) We show a critical scaling regime for the step-size, below which the effective ballistic dynamics matches gradient flow for the population loss. About the fixed points of this effective dynamics, the corresponding diffusive limits can be quite complex and even degenerate.
  arXiv  Detail & Related papers  (2022-06-08T17:42:18Z)
• Deep Learning Approximation of Diffeomorphisms via Linear-Control Systems [91.3755431537592]
  We consider a control system of the form $dot x = sum_i=1lF_i(x)u_i$, with linear dependence in the controls. We use the corresponding flow to approximate the action of a diffeomorphism on a compact ensemble of points.
  arXiv  Detail & Related papers  (2021-10-24T08:57:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.