Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime

• URL: http://arxiv.org/abs/2605.10931v1
• Date: Mon, 11 May 2026 17:58:14 GMT
• Title: Quantifying Concentration Phenomena of Mean-Field Transformers in the Low-Temperature Regime
• Abstract summary: We study the evolution of tokens in deep encoder-only transformers at inference time.<n>We prove that the token distribution rapidly concentrates onto the push-forward of the initial distribution.
• Score: 0.9974630621313314
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformers with self-attention modules as their core components have become an integral architecture in modern large language and foundation models. In this paper, we study the evolution of tokens in deep encoder-only transformers at inference time which is described in the large-token limit by a mean-field continuity equation. Leveraging ideas from the convergence analysis of interacting multi-particle systems, with particles corresponding to tokens, we prove that the token distribution rapidly concentrates onto the push-forward of the initial distribution under a projection map induced by the key, query, and value matrices, and remains metastable for moderate times. Specifically, we show that the Wasserstein distance of the two distributions scales like $\sqrt{{\log(β+1)}/β}\exp(Ct)+\exp(-ct)$ in terms of the temperature parameter $β^{-1}\to 0$ and inference time $t\geq 0$. For the proof, we establish Lyapunov-type estimates for the zero-temperature equation, identify its limit as $t\to\infty$, and employ a stability estimate in Wasserstein space together with a quantitative Laplace principle to couple the two equations. Our result implies that for time scales of order $\logβ$ the token distribution concentrates at the identified limiting distribution. Numerical experiments confirm this and, beyond that, complement our theory by showing that for finite $β$ and large $t$ the dynamics enter a different terminal phase, dominated by the spectrum of the value matrix.

Related papers

• Quantitative Target Convergence and Uniform-in-Time Propagation of Chaos for Langevin-Regularized SVGD [2.457924087844968]
  We show convergence to the target and uniform-in-time propagation of chaos for Langevin-regularized Stein descent.<n>For propagation of chaos, we develop two complementary finite-time approaches.
  arXiv  Detail & Related papers  (2026-08-28T19:56:49Z)
• Quantitative Convergence of Wasserstein Gradient Flows of Kernel Mean Discrepancies [10.511277414974613]
  We study the quantitative convergence of Wasserstein gradient flows of Kernel Mean Discrepancy functionals.<n>Our setting covers in particular the training dynamics of shallow neural networks in the infinite-width and continuous time limit.
  arXiv  Detail & Related papers  (2026-03-02T15:32:54Z)
• Stabilizing Fixed-Point Iteration for Markov Chain Poisson Equations [49.702772230127465]
  We study finite-state Markov chains with $n$ states and transition matrix $P$.<n>We show that all non-decaying modes are captured by a real peripheral invariant subspace $mathcalK(P)$, and that the induced operator on the quotient space $mathbbRn/mathcalK(P) is strictly contractive, yielding a unique quotient solution.
  arXiv  Detail & Related papers  (2026-01-31T02:57:01Z)
• Theta-term in Russian Doll Model: phase structure, quantum metric and BPS multifractality [45.88028371034407]
  We investigate the phase structure of the deterministic and disordered versions of the Russian Doll Model (RDM)<n>We find the pattern of phase transitions in the global charge $Q(theta,gamma)$, which arises from the BA equation.<n>We conjecture that the Hamiltonian of the RDM model describes the mixing in particular 2d-4d BPS sector of the Hilbert space.
  arXiv  Detail & Related papers  (2025-10-23T17:25:01Z)
• Improved Finite-Particle Convergence Rates for Stein Variational Gradient Descent [14.890609936348277]
  We provide finite-particle convergence rates for the Stein Variational Gradient Descent algorithm in the Kernelized Stein Discrepancy ($mathsfKSD$) and Wasserstein-2 metrics.<n>Our key insight is that the time derivative of the relative entropy between the joint density of $N$ particle locations splits into a dominant negative part' proportional to $N$ times the expected $mathsfKSD2$ and a smaller positive part'
  arXiv  Detail & Related papers  (2024-09-13T01:49:19Z)
• KPZ scaling from the Krylov space [83.88591755871734]
  Recently, a superdiffusion exhibiting the Kardar-Parisi-Zhang scaling in late-time correlators and autocorrelators has been reported. Inspired by these results, we explore the KPZ scaling in correlation functions using their realization in the Krylov operator basis.
  arXiv  Detail & Related papers  (2024-06-04T20:57:59Z)
• Quantum-critical properties of the one- and two-dimensional random transverse-field Ising model from large-scale quantum Monte Carlo simulations [0.0]
  We study the ferromagnetic transverse-field Ising model with quenched disorder at $T = 0$ in one and two dimensions. The emphasis on effective zero-temperature simulations resolves several inconsistencies in existing literature.
  arXiv  Detail & Related papers  (2024-03-08T11:20:42Z)
• Measurement-induced phase transition for free fermions above one dimension [46.176861415532095]
  Theory of the measurement-induced entanglement phase transition for free-fermion models in $d>1$ dimensions is developed. Critical point separates a gapless phase with $elld-1 ln ell$ scaling of the second cumulant of the particle number and of the entanglement entropy.
  arXiv  Detail & Related papers  (2023-09-21T18:11:04Z)
• Theory of free fermions under random projective measurements [43.04146484262759]
  We develop an analytical approach to the study of one-dimensional free fermions subject to random projective measurements of local site occupation numbers. We derive a non-linear sigma model (NLSM) as an effective field theory of the problem.
  arXiv  Detail & Related papers  (2023-04-06T15:19:33Z)
• Entanglement distribution in the Quantum Symmetric Simple Exclusion Process [0.0]
  We study the probability distribution of entanglement in the Quantum Symmetric Simple Exclusion Process. By means of a Coulomb gas approach from Random Matrix Theory, we compute analytically the large-deviation function of the entropy in the thermodynamic limit.
  arXiv  Detail & Related papers  (2021-02-09T10:25:04Z)
• Anisotropy-mediated reentrant localization [62.997667081978825]
  We consider a 2d dipolar system, $d=2$, with the generalized dipole-dipole interaction $sim r-a$, and the power $a$ controlled experimentally in trapped-ion or Rydberg-atom systems. We show that the spatially homogeneous tilt $beta$ of the dipoles giving rise to the anisotropic dipole exchange leads to the non-trivial reentrant localization beyond the locator expansion.
  arXiv  Detail & Related papers  (2020-01-31T19:00:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.