Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention

• URL: http://arxiv.org/abs/2605.04279v2
• Date: Fri, 08 May 2026 06:41:08 GMT
• Title: Gradient Flow Structure and Quantitative Dynamics of Multi-Head Self-Attention
• Abstract summary: Transformer self-attention can be interpreted as a gradient flow on the unit sphere.<n>We develop a theoretical framework for multi-head self-attention dynamics.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transformer self-attention can be interpreted as a gradient flow on the unit sphere, in which tokens evolve under softmax interaction potentials and tend to form clusters. While prior work has established clustering behavior for single-head attention, the multi-head setting remains less understood due to geometric interference between heads, which invalidates standard monotonicity arguments. In this work, we develop a theoretical framework for multi-head self-attention dynamics and resolve several open questions. We show that, under suitable conditions on the score matrices, a natural multi-head energy functional is non-decreasing along both flat and spherical dynamics. We identify the key obstruction to per-head monotonicity as radial shadow terms, which are projections of each head's output onto token directions, persisting even under orthogonality assumptions. We introduce a sufficient condition ensuring monotonicity and establish robustness to approximate orthogonality. In a simplified scalar-head regime with equiangular token configurations, we derive a closed-form expression for the critical inverse temperature governing clustering behavior, and show that heterogeneous heads exhibit super-additive clustering rates. In this regime, we also prove a separation in clustering time between ReLU and softmax attention in the linearized dynamics. Finally, we establish an entropy production identity and show that attention entropy increases monotonically toward equilibrium as clustering progresses. Our results provide a unified perspective on the dynamics of multi-head attention and clarify the mechanisms underlying clustering and stability in transformer models.

Related papers

• Clustered Attractor Manifolds and Dynamical Condensation in Self-Attention [2.4792581096761404]
  Transformer layers generate state-dependent interaction networks.<n> token representations determine the attention matrix, which in turn updates the representations.<n>We study this feedback in a minimal normalized self-attention dynamics.
  arXiv  Detail & Related papers  (2026-08-09T21:28:01Z)
• Neural Networks Provably Learn Spectral Representations for Group Composition [51.637084292404616]
  We study how structured internal structure emerges during neural network training.<n>We prove that the training dynamics are governed by a gradient ascent on a representation-theoretic energy functional.<n>This framework provides a representation-theoretic account of feature learning and characterizes a novel low-rank compression phenomenon.
  arXiv  Detail & Related papers  (2026-06-02T01:04:21Z)
• Homogenized Transformers [5.290251602267728]
  We study a random model of deep multi-head self-attention in which the weights are resampled independently across layers and heads.<n>We prove that, under suitable joint scalings of the depth, the residual step size, and the number of heads, this dynamics admits a nontrivialed limit.
  arXiv  Detail & Related papers  (2026-04-02T12:39:27Z)
• KoopGen: Koopman Generator Networks for Representing and Predicting Dynamical Systems with Continuous Spectra [65.11254608352982]
  We introduce a generator-based neural Koopman framework that models dynamics through a structured, state-dependent representation of Koopman generators.<n>By exploiting the intrinsic Cartesian decomposition into skew-adjoint and self-adjoint components, KoopGen separates conservative transport from irreversible dissipation.
  arXiv  Detail & Related papers  (2026-02-15T06:32:23Z)
• State Rank Dynamics in Linear Attention LLMs [37.607046806053035]
  State Rank Stratification is characterized by a distinct spectral bifurcation among linear attention heads.<n>Low-rank heads are indispensable for model reasoning, whereas high-rank heads exhibit significant redundancy.<n>We propose Joint Rank-Norm Pruning, a zero-shot strategy that achieves a 38.9% reduction in KV-cache overhead while largely maintaining model accuracy.
  arXiv  Detail & Related papers  (2026-02-02T15:00:42Z)
• The Geometric Mechanics of Contrastive Representation Learning: Alignment Potentials, Entropic Dispersion, and Cross-Modal Divergence [17.501700376593174]
  We present a measure-theoretic framework that models learning as the evolution of representation measures on a fixed embedding manifold.<n>By establishing value and consistency in the large-batch limit, we bridge the misalignment objective to explicit energy landscapes.<n>We show that this term induces barrier-driven co-adaptation, enforcing a population-level modality gap as a structural geometric necessity.
  arXiv  Detail & Related papers  (2026-01-27T13:33:03Z)
• Random-Matrix-Induced Simplicity Bias in Over-parameterized Variational Quantum Circuits [72.0643009153473]
  We show that expressive variational ansatze enter a Haar-like universality class in which both observable expectation values and parameter gradients concentrate exponentially with system size.<n>As a consequence, the hypothesis class induced by such circuits collapses with high probability to a narrow family of near-constant functions.<n>We further show that this collapse is not unavoidable: tensor-structured VQCs, including tensor-network-based and tensor-hypernetwork parameterizations, lie outside the Haar-like universality class.
  arXiv  Detail & Related papers  (2026-01-05T08:04:33Z)
• The Mean-Field Dynamics of Transformers [6.008788032203683]
  By idealizing attention on the sphere, we connect Transformer dynamics to Wasserstein gradient flows (Kuramoto), and mean-shift clustering.<n>Results highlight both the mechanisms that drive representation collapse and the regimes that preserve expressive, multi-cluster structure in deep attention architectures.
  arXiv  Detail & Related papers  (2025-12-01T16:51:00Z)
• RainDiff: End-to-end Precipitation Nowcasting Via Token-wise Attention Diffusion [64.49056527678606]
  We propose a Token-wise Attention integrated into not only the U-Net diffusion model but also the radar-temporal encoder.<n>Unlike prior approaches, our method integrates attention into the architecture without incurring the high resource cost typical of pixel-space diffusion.<n>Our experiments and evaluations demonstrate that the proposed method significantly outperforms state-of-the-art approaches, robustness local fidelity, generalization, and superior in complex precipitation forecasting scenarios.
  arXiv  Detail & Related papers  (2025-10-16T17:59:13Z)
• Kuramoto Orientation Diffusion Models [67.0711709825854]
  Orientation-rich images, such as fingerprints and textures, often exhibit coherent angular patterns.<n>Motivated by the role of phase synchronization in biological systems, we propose a score-based generative model.<n>We implement competitive results on general image benchmarks and significantly improves generation quality on orientation-dense datasets like fingerprints and textures.
  arXiv  Detail & Related papers  (2025-09-18T18:18:49Z)
• Training Dynamics of Multi-Head Softmax Attention for In-Context Learning: Emergence, Convergence, and Optimality [54.20763128054692]
  We study the dynamics of gradient flow for training a multi-head softmax attention model for in-context learning of multi-task linear regression. We prove that an interesting "task allocation" phenomenon emerges during the gradient flow dynamics.
  arXiv  Detail & Related papers  (2024-02-29T18:43:52Z)
• Dynamics of inhomogeneous spin ensembles with all-to-all interactions: breaking permutational invariance [49.1574468325115]
  We investigate the consequences of introducing non-uniform initial conditions in the dynamics of spin ensembles characterized by all-to-all interactions. We find that the dynamics of the spin ensemble now spans a more expansive effective Hilbert space.
  arXiv  Detail & Related papers  (2023-09-19T16:44:14Z)
• Driven-dissipative Ising Model: An exact field-theoretical analysis [0.0]
  Driven-dissipative many-body systems are difficult to analyze analytically due to their non-equilibrium dynamics, dissipation and many-body interactions. We develop an exact field-theoretical analysis and a diagrammatic representation of the spin model that can be understood from a simple scattering picture.
  arXiv  Detail & Related papers  (2021-01-13T19:00:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.