On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers

• URL: http://arxiv.org/abs/2607.18584v1
• Date: Mon, 20 Jul 2026 23:43:26 GMT
• Title: On the Diverse Dynamical Behaviors Arising in Deep Linear Transformers
• Abstract summary: We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems.<n>We show that the dynamics can be reformulated as a generalized Kuramoto-type model with pure second-harmonic coupling.
• Score: 34.73438474762916
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study the inference-time behavior of deep linear encoder-only transformers through the lens of interacting particle systems. In this perspective, tokens are modeled as particles that interact dynamically through successive linear self-attention layers. We show that in embedding dimension two, for any key, query, and value matrices, the dynamics can be reformulated as a generalized Kuramoto-type model with pure second-harmonic coupling. This formulation is amenable to Watanabe--Strogatz theory which reveals the dynamics are intrinsically low-dimensional regardless of the parameter matrices. For a class of token initializations associated with the Ott--Antonsen (OA) manifold, we show that the parameter matrices induce a diverse variety of long-time behaviors in linear transformers, including clustering, oscillations, and bifurcations. The oscillations and bifurcations are characterized by uncovering a hidden Hamiltonian structure in the dynamics. By establishing a structural stability result, we further show that dynamics initialized near the OA manifold exhibit the same long-time behavior as those initialized exactly on the manifold. Motivated by our theory in dimension two, we conduct numerical experiments for analogous parameter regimes in higher-dimensional transformers. Our numerical experiments suggest that the long-time behaviors characterized in our theoretical results persist in higher dimensions.

Related papers

• Multibath Influence Matrices: Universal Scaling from Real-Time Dynamics [0.005023988093549015]
  Diverging timescales are the hallmark of critical dynamics and the bottleneck to their classical and quantum simulation.<n>We compress a spacetime tensor network temporally into semigroup influence matrices and spatially via matrix-product states.<n>Together, these results establish multibath influence matrices as a practical tool for real-time dynamics in strongly correlated multi-orbital systems.
  arXiv  Detail & Related papers  (2026-07-17T18:04:36Z)
• Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks [50.79340930913805]
  We study a class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning.<n>By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.
  arXiv  Detail & Related papers  (2026-07-13T17:56:11Z)
• Emergent Manifold Separability during Reasoning in Large Language Models [46.78826734548872]
  Chain-of-Thought prompting significantly improves reasoning in Large Language Models.<n>We quantify the linear separability of latent representations without the confounding factors of probe training.
  arXiv  Detail & Related papers  (2026-02-23T20:36:17Z)
• Multicriticality in stochastic dynamics protected by self-duality [0.0]
  We study the large dynamical deviations (LD) of a class of one-dimensional kinetically constrained models.<n>We consider four representative models in detail: the domain-wall (DW) Fredrickson-Andersen (FA), the DW East, the ZZZ-FA, and the XOR-FA models.
  arXiv  Detail & Related papers  (2025-04-02T00:10:57Z)
• Nonlinear Dynamics of a Dicke Model for V-Type Atoms [0.0]
  We study the nonlinear, semiclassical dynamics of an open spin-1 (three-level) variant of the traditional Dicke model.<n>We find the emergence of both periodic and two-frequency oscillations, as well as multistability and chaotic dynamics.
  arXiv  Detail & Related papers  (2025-03-12T21:35:14Z)
• Dyson Brownian motion and random matrix dynamics of weight matrices during learning [0.0]
  We first demonstrate that the dynamics can generically be described using Dyson Brownian motion. The level ofity is shown to depend on the ratio of the learning rate and the mini-batch size. We then study weight matrix dynamics in transformers following the evolution from a Marchenko-Pastur distribution for eigenvalues at initialisation to a combination with additional structure at the end of learning.
  arXiv  Detail & Related papers  (2024-11-20T18:05:39Z)
• Universal Approximation of Mean-Field Models via Transformers [3.8916312075738273]
  This paper investigates the use of transformers to approximate the mean-field dynamics of interacting particle systems exhibiting collective behavior.<n>Such systems are fundamental in modeling phenomena across physics, biology, and engineering, including opinion formation, biological networks, and swarm robotics.
  arXiv  Detail & Related papers  (2024-10-06T19:47:24Z)
• Visualizing Dynamics of Charges and Strings in (2+1)D Lattice Gauge Theories [103.95523007319937]
  Lattice gauge theories (LGTs) can be employed to understand a wide range of phenomena.<n>Here, we investigate the dynamics of local excitations in a $mathZ$ LGT using a two-dimensional lattice of superconducting qubits.<n>Our method allows us to experimentally image string dynamics in a (2+1)D LGT from which we uncover two distinct regimes.
  arXiv  Detail & Related papers  (2024-09-25T17:59:05Z)
• Latent Space Energy-based Neural ODEs [73.01344439786524]
  This paper introduces novel deep dynamical models designed to represent continuous-time sequences.<n>We train the model using maximum likelihood estimation with Markov chain Monte Carlo.<n> Experimental results on oscillating systems, videos and real-world state sequences (MuJoCo) demonstrate that our model with the learnable energy-based prior outperforms existing counterparts.
  arXiv  Detail & Related papers  (2024-09-05T18:14:22Z)
• Nonlinear dynamical Casimir effect and Unruh entanglement in waveguide QED with parametrically modulated coupling [83.88591755871734]
  We study theoretically an array of two-level qubits moving relative to a one-dimensional waveguide. When the frequency of this motion approaches twice the qubit resonance frequency, it induces parametric generation of photons and excitation of the qubits. We develop a comprehensive general theoretical framework that incorporates both perturbative diagrammatic techniques and a rigorous master-equation approach.
  arXiv  Detail & Related papers  (2024-08-30T15:54:33Z)
• Dynamics of a Generalized Dicke Model for Spin-1 Atoms [0.0]
  The Dicke model is a staple of theoretical cavity Quantum Electrodynamics (cavity QED) It demonstrates a rich variety of dynamics such as phase transitions, phase multistability, and chaos. The varied and complex behaviours admitted by the model highlights the need to more rigorously map its dynamics.
  arXiv  Detail & Related papers  (2024-03-04T04:09:35Z)
• Generating function for projected entangled-pair states [0.1759252234439348]
  We extend the generating function approach for tensor network diagrammatic summation. Taking the form of a one-particle excitation, we show that the excited state can be computed efficiently in the generating function formalism. We conclude with a discussion on generalizations to multi-particle excitations.
  arXiv  Detail & Related papers  (2023-07-16T15:49:37Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.