Reachability and asymptotics of Gaussian Transformer dynamics

• URL: http://arxiv.org/abs/2606.07600v1
• Date: Fri, 29 May 2026 07:51:03 GMT
• Title: Reachability and asymptotics of Gaussian Transformer dynamics
• Abstract summary: We formulate data propagation through the Transformer, the machine learning architecture powering large language models.<n>We prove that Gaussian distributions remain exactly Gaussian along the induced flow.<n>We reformulates the expressive capacity of Transformers as a reachability problem for prescribed Gaussian moments.
• Score: 2.669398324965473
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We formulate data propagation through the Transformer, the machine learning architecture powering large language models, as a nonlinear control system on the space of probability measures. For the mean-field Transformer model with self-attention and affine feed-forward layers, we prove that Gaussian distributions remain exactly Gaussian along the induced flow. This invariance reduces the infinite-dimensional measure dynamics to a finite-dimensional bilinear control system governing the evolution of the mean and covariance, reformulates the expressive capacity of Transformers as a reachability problem for prescribed Gaussian moments, and reveals a novel connection with Riccati-type equations from classical filtering and control. For time-varying controls, we prove exact finite-time reachability of any target Gaussian distribution whose covariance matrix has the same rank as the initial one, this rank constraint being an intrinsic invariant of the dynamics. For time-invariant parameters, we derive explicit spectral conditions leading either to asymptotic stability toward positive-definite equilibria or to finite-time blow-up of the covariance. Numerical experiments complement the theory by showing that practical Transformers with Gaussian inputs remain close to moment-matched Gaussian distributions through early and intermediate layers, while Transformers with prescribed attention matrices reproduce the predicted covariance regimes: bounded evolution in stabilizing configurations and blow-up in destabilizing ones.

Related papers

• Generalization Bounds on Optimal Control for Transformer Training and Wasserstein Distributional Robustness [8.564319625930894]
  We view data sets as probability laws on pairs of empirical input-output measures, allowing us to interpret the training problem as a finite-horizon Markovian control problem.<n>We show that the same machinery yields a distributionally robust control formulation of the training problem, connecting Transformer generalization to Wasserstein distributionally robust optimization.
  arXiv  Detail & Related papers  (2026-07-30T10:22:38Z)
• Stability and Generalization of Push-Sum Based Decentralized Optimization over Directed Graphs [55.77845440440496]
  Push-based decentralized communication enables optimization over communication networks, where information exchange may be asymmetric.<n>We develop a unified uniform-stability framework for the Gradient Push (SGP) algorithm.<n>A key technical ingredient is an imbalance-aware generalization bound through two quantities.
  arXiv  Detail & Related papers  (2026-02-24T05:32:03Z)
• Rethinking Diffusion Models with Symmetries through Canonicalization with Applications to Molecular Graph Generation [56.361076943802594]
  CanonFlow achieves state-of-the-art performance on the challenging GEOM-DRUG dataset, and the advantage remains large in few-step generation.
  arXiv  Detail & Related papers  (2026-02-16T18:58:55Z)
• Diffusion-Inspired Reconfiguration of Transformers for Uncertainty Calibration [52.017716672255524]
  Uncertainty calibration in pre-trained transformers is critical for their reliable deployment in risk-sensitive applications.<n>We propose a diffusion-inspired reconfiguration of transformers in which each feature transformation block is modeled as a probabilistic mapping.<n>Our method achieves superior calibration and predictive accuracy compared to existing uncertainty-aware transformers.
  arXiv  Detail & Related papers  (2026-02-09T17:24:47Z)
• Random-Matrix-Induced Simplicity Bias in Over-parameterized Variational Quantum Circuits [72.0643009153473]
  We show that expressive variational ansatze enter a Haar-like universality class in which both observable expectation values and parameter gradients concentrate exponentially with system size.<n>As a consequence, the hypothesis class induced by such circuits collapses with high probability to a narrow family of near-constant functions.<n>We further show that this collapse is not unavoidable: tensor-structured VQCs, including tensor-network-based and tensor-hypernetwork parameterizations, lie outside the Haar-like universality class.
  arXiv  Detail & Related papers  (2026-01-05T08:04:33Z)
• Closed-Loop Transformers: Autoregressive Modeling as Iterative Latent Equilibrium [0.6820746164515952]
  We introduce the closed-loop prediction principle, which requires that models iteratively refine latent representations until reaching a self-consistent equilibrium.<n>We instantiate this principle as Equilibrium Transformers, which augment standard transformer layers with an Equilibrium Refinement Module.<n>Preliminary experiments on the binary parity task demonstrate +3.28% average improvement on challenging sequences, with gains reaching +8.07% where standard transformers approach random performance.
  arXiv  Detail & Related papers  (2025-11-26T20:02:59Z)
• Scalable Equilibrium Sampling with Sequential Boltzmann Generators [60.00515282300297]
  We extend the Boltzmann generator framework with two key contributions.<n>The first is a highly efficient Transformer-based normalizing flow operating directly on all-atom Cartesian coordinates.<n>In particular, we perform inference-time scaling of flow samples using a continuous-time variant of sequential Monte Carlo.
  arXiv  Detail & Related papers  (2025-02-25T18:59:13Z)
• Asymptotics of Linear Regression with Linearly Dependent Data [28.005935031887038]
  We study the computations of linear regression in settings with non-Gaussian covariates.<n>We show how dependencies influence estimation error and the choice of regularization parameters.
  arXiv  Detail & Related papers  (2024-12-04T20:31:47Z)
• Geometric Dynamics of Signal Propagation Predict Trainability of Transformers [22.25628914395565]
  We investigate forward signal propagation and gradient back propagation in deep, randomly transformers. Our approach treats the evolution of $n tokens as they propagate through the transformer layers. We show through experiments that, remarkably, the final test loss at the end of training is well predicted just by these two exponents.
  arXiv  Detail & Related papers  (2024-03-05T01:30:34Z)
• Machine learning in and out of equilibrium [58.88325379746631]
  Our study uses a Fokker-Planck approach, adapted from statistical physics, to explore these parallels. We focus in particular on the stationary state of the system in the long-time limit, which in conventional SGD is out of equilibrium. We propose a new variation of Langevin dynamics (SGLD) that harnesses without replacement minibatching.
  arXiv  Detail & Related papers  (2023-06-06T09:12:49Z)
• NAG-GS: Semi-Implicit, Accelerated and Robust Stochastic Optimizer [45.47667026025716]
  We propose a novel, robust and accelerated iteration that relies on two key elements. The convergence and stability of the obtained method, referred to as NAG-GS, are first studied extensively. We show that NAG-arity is competitive with state-the-art methods such as momentum SGD with weight decay and AdamW for the training of machine learning models.
  arXiv  Detail & Related papers  (2022-09-29T16:54:53Z)
• Signatures of Chaos in Non-integrable Models of Quantum Field Theory [0.0]
  We study signatures of quantum chaos in (1+1)D Quantum Field Theory (QFT) models. We focus on the double sine-Gordon, also studying the massive sine-Gordon and $phi4$ model.
  arXiv  Detail & Related papers  (2020-12-15T18:56:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.