Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models

• URL: http://arxiv.org/abs/2604.26898v1
• Date: Wed, 29 Apr 2026 17:09:05 GMT
• Title: Stochastic Scaling Limits and Synchronization by Noise in Deep Transformer Models
• Abstract summary: We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer with Multi-Layer Perceptron (MLP) blocks.<n>We also identify the partial differential equation describing the evolution of the tokens' distribution in this limit and prove propagation of chaos when the number of such tokens is large.
• Score: 6.60601600774899
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We prove pathwise convergence of the layerwise evolution of tokens in a finite-depth, finite-width transformer model with MultiLayer Perceptron (MLP) blocks to a continuous-time stochastic interacting particle system. We also identify the stochastic partial differential equation describing the evolution of the tokens' distribution in this limit and prove propagation of chaos when the number of such tokens is large. The bounds we establish are quantitative and the limits we consider commute. We further prove that the limiting stochastic model displays synchronization by noise and establish exponential dissipation of the interaction energy on average, provided that the common noise is sufficiently coercive relative to the deterministic self-attention drift. We finally characterize the activation functions satisfying the former condition.

Related papers

• Frequency-Aware Flow Matching for Continuous and Consistent Robotic Action Generation [46.998628831231194]
  Existing methods rely on discretized action chunks, making them brittle to demonstrations collected at heterogeneous control.<n>We propose Frequency-Aware Flow Matching (FAFM), which outputs continuous, temporally consistent actions.<n>FAFM improves success rates, multimodal expressivity, motion smoothness, convergence speed, robustness to mechanical bias and mixed-frequency input.
  arXiv  Detail & Related papers  (2026-06-18T11:58:30Z)
• Quantum-to-Classical Computability Transition via Negative Markov Chains [0.0]
  We show that for any unitary evolution generated by a linear combination of local or pairwise interactions, there exists at least one noise channel that effectively classicalizes the system.<n>We show that for this class of unitaries, the dynamics of an open quantum spin chain subject to depolarizing noise undergoes an exact transition to classical simulability.
  arXiv  Detail & Related papers  (2026-04-21T18:11:19Z)
• Homogenized Transformers [5.290251602267728]
  We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads.<n>We prove that, under suitable joint scalings of the depth, the residual step size, and the number of heads, this dynamics admits a nontrivialed limit.
  arXiv  Detail & Related papers  (2026-04-02T12:39:27Z)
• Parallel Complex Diffusion for Scalable Time Series Generation [50.01609741902786]
  PaCoDi is a spectral-native architecture that decouples generative modeling in the frequency domain.<n>We show that PaCoDi outperforms existing baselines in both generation quality and inference speed.
  arXiv  Detail & Related papers  (2026-02-10T14:31:53Z)
• Clustering in Deep Stochastic Transformers [10.988655177671255]
  Existing theories of deep Transformers with layer normalization typically predict that tokens cluster to a single point.<n>We analyze deep Transformers where noise arises from the random value of value.<n>For two tokens, we prove a phase transition governed by the interaction strength and the token dimension.
  arXiv  Detail & Related papers  (2026-01-29T16:28:13Z)
• Propagation of Chaos in One-hidden-layer Neural Networks beyond Logarithmic Time [46.15741640288809]
  We study the approximation gap between the dynamics of a-width neural network and its infinite-width counterpart.<n>We demonstrate how to tightly bound this approximation gap through a differential equation governed by the mean-field dynamics.
  arXiv  Detail & Related papers  (2025-04-17T17:24:38Z)
• Dynamical transitions from slow to fast relaxation in random open quantum systems [0.0]
  We study a model in which the system Hamiltonian and its couplings to the noise are random matrices whose entries decay as power laws of distance. The steady state is always featureless, but the rate at which it is approached exhibits three phases depending on $alpha_H$ and $alpha_L$. Within perturbation theory, the phase boundaries in the $(alpha_H, alpha_L)$ plane differ for weak and strong dissipation, suggesting phase transitions as a function of noise strength.
  arXiv  Detail & Related papers  (2022-11-23T20:56:46Z)
• Decimation technique for open quantum systems: a case study with driven-dissipative bosonic chains [62.997667081978825]
  Unavoidable coupling of quantum systems to external degrees of freedom leads to dissipative (non-unitary) dynamics. We introduce a method to deal with these systems based on the calculation of (dissipative) lattice Green's function. We illustrate the power of this method with several examples of driven-dissipative bosonic chains of increasing complexity.
  arXiv  Detail & Related papers  (2022-02-15T19:00:09Z)
• Entanglement Transitions from Stochastic Resetting of Non-Hermitian Quasiparticles [0.0]
  We write down a renewal equation for the statistics of the entanglement entropy and show that depending on the spectrum of quasiparticle decay rates different entanglement scaling can arise and even sharp entanglement phase transitions. When applied to a Quantum Ising chain where the transverse magnetization is measured by quantum jumps, our theory predicts a critical phase with logarithmic scaling of the entanglement, an area law phase and a continuous phase transition between them, with an effective central charge vanishing as a square root at the transition point.
  arXiv  Detail & Related papers  (2021-11-05T13:38:04Z)
• Continuous-time dynamics and error scaling of noisy highly-entangling quantum circuits [58.720142291102135]
  We simulate a noisy quantum Fourier transform processor with up to 21 qubits. We take into account microscopic dissipative processes rather than relying on digital error models. We show that depending on the dissipative mechanisms at play, the choice of input state has a strong impact on the performance of the quantum algorithm.
  arXiv  Detail & Related papers  (2021-02-08T14:55:44Z)
• From stochastic spin chains to quantum Kardar-Parisi-Zhang dynamics [68.8204255655161]
  We introduce the asymmetric extension of the Quantum Symmetric Simple Exclusion Process. We show that the time-integrated current of fermions defines a height field which exhibits a quantum non-linear dynamics.
  arXiv  Detail & Related papers  (2020-01-13T14:30:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.