Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows

• URL: http://arxiv.org/abs/2605.18870v1
• Date: Fri, 15 May 2026 15:32:18 GMT
• Title: Multi-Headed Transformer Architectures as Time-dependent Wasserstein Gradient Flows
• Abstract summary: We study the data flow in multi-headed transformer architectures as time-dependent gradient flows for a suitable interaction energy.<n>We prove the $$-convergence of the perturbed interaction energy to the unperturbed one, leading to the convergence of the corresponding gradient flows.<n>We complement these theoretical results with numerical experiments that confirm the predicted energy-dissipation identity and clarify the behavior of the dynamics in both the autonomous-like (Ornstein-Uhlenbeck) and the genuinely non-autonomous (oscillating-weights) regimes.
• Score: 16.7218879608165
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In recent years, transformer architectures have revolutionized the field of language processing, opening the door to previously unforeseen possibilities. However, from a theoretical point of view, the mathematical models proposed in the literature often lack direct contact with the actual architectures and depend on strong simplifying assumptions. In this paper, we reduce this gap by modelling the data flow in multi-headed transformer architectures as time-dependent gradient flows for a suitable interaction energy capturing the design of the attention mechanism. The explicit dependence on time allows us to consider different weights for each head and for each layer, without imposing constraints on the initialization method. Moreover, we prove that, under a suitable integrability assumption on the evolution of the weights, each element of the $ω$-limit set of the gradient flows is a stationary point of the interaction energy at a limiting weight distribution. Finally, we analyse the stability of the gradient flows considering perturbations of both the initial data and the weights. Specifically, on the one hand, we study the robustness of the proposed models with respect to noisy inputs, establishing a continuous dependence of the gradient flows on the initial data and uniqueness of the flows. On the other hand, we prove the $Γ$-convergence of the perturbed interaction energy to the unperturbed one, leading to the convergence of the corresponding gradient flows. We complement these theoretical results with numerical experiments that confirm the predicted energy-dissipation identity and clarify the asymptotic behavior of the dynamics in both the autonomous-like (Ornstein--Uhlenbeck) and the genuinely non-autonomous (oscillating-weights) regimes.

Related papers

• Scaling at Chiral Clock Criticality via Entanglement Renormalization [0.12277343096128711]
  We employ the Multiscale Entanglement Renormalization Ansatz (MERA) network to investigate the $mathbbZ_3$ chiral clock model.<n>We find a set of effective scaling data that smoothly varies starting from the Potts data as the chiral parameter is increased.<n>Our findings demonstrate MERA's effectiveness in capturing the complex low-energy physics of the chiral clock model.
  arXiv  Detail & Related papers  (2026-04-21T18:00:22Z)
• Symmetry-protected topology and deconfined solitons in a multi-link $\mathbb{Z}_2$ gauge theory [45.88028371034407]
  We study a $mathbbZ$ lattice gauge theory defined on a multi-graph with links that can be visualized as great circles of a spherical shell.<n>We show that this leads to state-dependent tunneling amplitudes underlying a phenomenon analogous to the Peierls instability.<n>By performining a detailed analysis based on matrix product states, we prove that charge deconfinement emerges as a consequence of charge-fractionalization.
  arXiv  Detail & Related papers  (2026-03-02T22:59:25Z)
• Plug-and-Play Diffusion Meets ADMM: Dual-Variable Coupling for Robust Medical Image Reconstruction [45.25461515976432]
  Plug-and-Play diffusion prior (DP) frameworks have emerged as a powerful paradigm for imaging reconstruction.<n>We present a novel approach to resolving bias-hallucination trade-off, achieving state-of-the-art gradients with significantly accelerated convergence.
  arXiv  Detail & Related papers  (2026-02-26T16:58:43Z)
• Emergent Manifold Separability during Reasoning in Large Language Models [46.78826734548872]
  Chain-of-Thought prompting significantly improves reasoning in Large Language Models.<n>We quantify the linear separability of latent representations without the confounding factors of probe training.
  arXiv  Detail & Related papers  (2026-02-23T20:36:17Z)
• Backpropagation as Physical Relaxation: Exact Gradients in Finite Time [0.0]
  ''Dyadic Backpropagation'' is the foundational algorithm for training neural networks.<n>We show it emerges exactly as the finite-time relaxation of a physical dynamical system.<n>We prove that unit-step Euler discretization, the natural timescale of layer transitions, recovers standard backpropagation exactly in precisely 2L steps.
  arXiv  Detail & Related papers  (2026-02-02T16:21:05Z)
• Latent Object Permanence: Topological Phase Transitions, Free-Energy Principles, and Renormalization Group Flows in Deep Transformer Manifolds [0.5729426778193398]
  We study the emergence of multi-step reasoning in deep Transformer language models through a geometric and statistical-physics lens.<n>We formalize the forward pass as a discrete coarse-graining map and relate the appearance of stable "concept basins" to fixed points of this renormalization-like dynamics.<n>The resulting low-entropy regime is characterized by a spectral tail collapse and by the formation of transient, reusable object-like structures in representation space.
  arXiv  Detail & Related papers  (2026-01-16T23:11:02Z)
• HFNO: an interpretable data-driven decomposition strategy for turbulent flows [0.0]
  We present a novel FNO-based architecture tailored for reduced-order modeling of turbulent fluid flows.<n>The proposed architecture processes wavenumber bins in parallel, enabling approximation of dispersion relations and non-linear interactions.<n>We evaluate the proposed model on a series of increasingly complex dynamical systems.
  arXiv  Detail & Related papers  (2025-11-03T12:57:19Z)
• Particle Dynamics for Latent-Variable Energy-Based Models [12.84928511163926]
  Latent-variable energy-based models (LVEBMs) assign a single normalized energy to joint pairs of observed data and latent variables.<n>We recast maximum-likelihood training as a saddle problem over distributions on the latent and joint gradients.<n>We prove existence and convergence under standard smoothness and dissipativity assumptions, with decay rates in KL divergence and Wasserstein-2 distance.
  arXiv  Detail & Related papers  (2025-10-17T09:04:49Z)
• TANGO: Time-Reversal Latent GraphODE for Multi-Agent Dynamical Systems [43.39754726042369]
  We propose a simple-yet-effective self-supervised regularization term as a soft constraint that aligns the forward and backward trajectories predicted by a continuous graph neural network-based ordinary differential equation (GraphODE) It effectively imposes time-reversal symmetry to enable more accurate model predictions across a wider range of dynamical systems under classical mechanics. Experimental results on a variety of physical systems demonstrate the effectiveness of our proposed method.
  arXiv  Detail & Related papers  (2023-10-10T08:52:16Z)
• Initial Correlations in Open Quantum Systems: Constructing Linear Dynamical Maps and Master Equations [62.997667081978825]
  We show that, for any predetermined initial correlations, one can introduce a linear dynamical map on the space of operators of the open system. We demonstrate that this construction leads to a linear, time-local quantum master equation with generalized Lindblad structure.
  arXiv  Detail & Related papers  (2022-10-24T13:43:04Z)
• Probing eigenstate thermalization in quantum simulators via fluctuation-dissipation relations [77.34726150561087]
  The eigenstate thermalization hypothesis (ETH) offers a universal mechanism for the approach to equilibrium of closed quantum many-body systems. Here, we propose a theory-independent route to probe the full ETH in quantum simulators by observing the emergence of fluctuation-dissipation relations. Our work presents a theory-independent way to characterize thermalization in quantum simulators and paves the way to quantum simulate condensed matter pump-probe experiments.
  arXiv  Detail & Related papers  (2020-07-20T18:00:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.