Homogenized Transformers

• URL: http://arxiv.org/abs/2604.01978v1
• Date: Thu, 02 Apr 2026 12:39:27 GMT
• Title: Homogenized Transformers
• Abstract summary: We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads.<n>We prove that, under suitable joint scalings of the depth, the residual step size, and the number of heads, this dynamics admits a nontrivialed limit.
• Score: 5.290251602267728
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads, as at initialization of training. Viewing depth as a time variable, the residual stream defines a discrete-time interacting particle system on the unit sphere. We prove that, under suitable joint scalings of the depth, the residual step size, and the number of heads, this dynamics admits a nontrivial homogenized limit. Depending on the scaling, the limit is either deterministic or stochastic with common noise; in the mean-field regime, the latter leads to a stochastic nonlinear Fokker--Planck equation for the conditional law of a representative token. In the Gaussian setting, the limiting drift vanishes, making the homogenized dynamics explicit enough to study representation collapse. This yields quantitative trade-offs between dimension, context length, and temperature, and identifies regimes in which clustering can be mitigated.

Related papers

• Monitored free fermions under periodic driving [42.078441758883635]
  We investigate periodically driven free-fermionic systems subjected to monitoring of the local particle density.<n>We use the nonlinear sigma-model (NLSM) to explain the long-wavelength physics of the time-dependent Hamiltonian.<n>Our results provide a unified analytical and numerical framework for understanding the effects of monitoring in time-modulated fermionic systems.
  arXiv  Detail & Related papers  (2026-08-22T08:10:12Z)
• A Mean-Field Analysis of Multi-Head Self-Attention under Cross-Entropy Training [0.0]
  This paper develops a mean-field theory for a single-layer causal multi-head self-attention model trained by cross-entropy minimization.<n>In the infinite-head limit, the averaged attention logits define a risk functional on probability measures, whose first variation generates a nonlinear Wasserstein gradient-flow equation.<n>We study the long-time behavior of the PDE: energy dissipation, convergence to the stationary set under compactness, convergence to a single stationary measure under topological or Kurdyka--ojasiewicz assumptions, and explicit convergence rates under gradient-domination conditions.
  arXiv  Detail & Related papers  (2026-06-09T06:38:27Z)
• Why SGD is not Brownian Motion: A New Perspective on Stochastic Dynamics [92.39053980710702]
  Gradient Descent (SGD) is commonly modeled as a Langevin process, assuming that minibatch noise acts as Brownian motion.<n>This approximation relies on a continuous-time limit and a sqrt(eta) noise scaling that does not match the discrete SGD update at finite learning rate.<n>We propose an alternative formulation of SGD as deterministic dynamics in a fluctuating loss landscape induced by minibatch sampling.
  arXiv  Detail & Related papers  (2026-05-21T15:50:40Z)
• Uniform Scaling Limits in AdamW-Trained Transformers [0.21485350418225238]
  We study the large-depth limit of transformers trained with AdamW.<n>We prove that the joint dynamics of the hidden states and backpropagated variables converge in $L2$.
  arXiv  Detail & Related papers  (2026-05-11T16:54:46Z)
• Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models [6.60601600774899]
  We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer with Multi-Layer Perceptron (MLP) blocks.<n>We also identify the partial differential equation describing the evolution of the tokens' distribution in this limit and prove propagation of chaos when the number of such tokens is large.
  arXiv  Detail & Related papers  (2026-04-29T17:09:05Z)
• Symmetry-protected topology and deconfined solitons in a multi-link $\mathbb{Z}_2$ gauge theory [45.88028371034407]
  We study a $mathbbZ$ lattice gauge theory defined on a multi-graph with links that can be visualized as great circles of a spherical shell.<n>We show that this leads to state-dependent tunneling amplitudes underlying a phenomenon analogous to the Peierls instability.<n>By performining a detailed analysis based on matrix product states, we prove that charge deconfinement emerges as a consequence of charge-fractionalization.
  arXiv  Detail & Related papers  (2026-03-02T22:59:25Z)
• Latent-Variable Learning of SPDEs via Wiener Chaos [2.0901018134712297]
  We study the problem of learning the law of linear partial differential equations (SPDEs) with additive Gaussian forcing from observations.<n>Our approach combines a spectral Galerkin projection with a truncated Wiener chaos expansion, yielding a separation between evolution and forcing domains.<n>This reduces the infinite-dimensional deterministic SPDE to a finite system of parametrized ordinary differential equations governing latent temporal dynamics.
  arXiv  Detail & Related papers  (2026-02-12T10:19:43Z)
• Emergence of Distortions in High-Dimensional Guided Diffusion Models [11.774563966512707]
  We formalize the phenomenon of generative distortion defined as the mismatch between the CFG-induced sampling and the true conditional distribution.<n>We show that standard CFG schedules are incapable of preventing variance shrinkage.<n>We propose a theoretically motivated guidance schedule featuring a negative-guidance window, which mitigates loss of diversity while preserving class separability.
  arXiv  Detail & Related papers  (2026-01-31T13:19:45Z)
• Chaos, Entanglement and Measurement: Field-Theoretic Perspectives on Quantum Information Dynamics [0.0]
  I study scrambling and pseudorandomness in the Brownian Sachdev-Ye-Kitaev (SYK) model.<n>I construct a field theory for weakly measured SYK clusters.<n>I develop a strong-disorder renormalization group for measurement-only SYK clusters.
  arXiv  Detail & Related papers  (2025-12-11T10:04:30Z)
• Preconditioned Regularized Wasserstein Proximal Sampling [2.7957842724446174]
  We consider sampling from a noise-free distribution by evolving finitely many particles.<n>For potentials, we provide a non-asymotic convergence analysis and explicitly the bias, which is dependent on regularization.
  arXiv  Detail & Related papers  (2025-09-01T18:04:31Z)
• Spin-only dynamics of the multi-species nonreciprocal Dicke model [0.0]
  Hepp-Lieb-Dicke model is ubiquitous in cavity quantum electrodynamics.<n>We study a variation of the open Dicke model which realizes mediated nonreciprocal interactions between spin species.<n>We find signatures of phase transitions even for small system sizes.
  arXiv  Detail & Related papers  (2025-07-10T17:41:46Z)
• Propagation of Chaos in One-hidden-layer Neural Networks beyond Logarithmic Time [46.15741640288809]
  We study the approximation gap between the dynamics of a-width neural network and its infinite-width counterpart.<n>We demonstrate how to tightly bound this approximation gap through a differential equation governed by the mean-field dynamics.
  arXiv  Detail & Related papers  (2025-04-17T17:24:38Z)
• Global Convergence of Over-parameterized Deep Equilibrium Models [52.65330015267245]
  A deep equilibrium model (DEQ) is implicitly defined through an equilibrium point of an infinite-depth weight-tied model with an input-injection. Instead of infinite computations, it solves an equilibrium point directly with root-finding and computes gradients with implicit differentiation. We propose a novel probabilistic framework to overcome the technical difficulty in the non-asymptotic analysis of infinite-depth weight-tied models.
  arXiv  Detail & Related papers  (2022-05-27T08:00:13Z)
• Determination of the critical exponents in dissipative phase transitions: Coherent anomaly approach [51.819912248960804]
  We propose a generalization of the coherent anomaly method to extract the critical exponents of a phase transition occurring in the steady-state of an open quantum many-body system.
  arXiv  Detail & Related papers  (2021-03-12T13:16:18Z)
• The role of boundary conditions in quantum computations of scattering observables [58.720142291102135]
  Quantum computing may offer the opportunity to simulate strongly-interacting field theories, such as quantum chromodynamics, with physical time evolution. As with present-day calculations, quantum computation strategies still require the restriction to a finite system size. We quantify the volume effects for various $1+1$D Minkowski-signature quantities and show that these can be a significant source of systematic uncertainty.
  arXiv  Detail & Related papers  (2020-07-01T17:43:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.