Kuramoto Attention: Synchronizing Self-Attention on the Torus

• URL: http://arxiv.org/abs/2606.11585v1
• Date: Wed, 10 Jun 2026 02:24:04 GMT
• Title: Kuramoto Attention: Synchronizing Self-Attention on the Torus
• Abstract summary: We introduce Kuramoto attention, a self-attention layer in which each hidden coordinate is an angle.<n>The layer scores tokens by gated cosine similarity, attends over previous phase states, and updates each token by the tangent component of the attention-weighted circular mean.<n>On enwiki8 character-level language modeling, the layer trains as a functional language model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce Kuramoto attention, a self-attention layer in which each hidden coordinate is an angle. The layer scores tokens by gated cosine similarity, attends over previous phase states, and updates each token by the tangent component of the attention-weighted circular mean. Because the values are the raw phase states, this update is exactly the Kuramoto coupling term $\sum_u A_{t,u}\sin(θ_u-θ_t)$, with the attention matrix acting as an adaptive, content-dependent coupling kernel. Equivalently, the gated score is a learned metric on the torus that selects which tokens couple, and the update pulls each token toward the circular mean of the tokens it selects, tightening their phase agreement. The same two ingredients, an invariant similarity score and an on-manifold mean, define such a layer on any compact group; the torus is the abelian case, where both are closed-form. The softmax weights solve an entropy-regularized phase-retrieval problem, and rotary position enters as a position-dependent phase drift in the score. On enwiki8 character-level language modeling, the layer trains as a functional language model whose bits-per-character stays close to a strong matched RoPE+SwiGLU transformer: within $0.02$ BPC at one million parameters ($1.637\pm0.010$ versus $1.616\pm0.004$) and level on the median at five million ($1.448$ versus $1.452$ over five seeds) with the transformer ahead on the mean ($1.468$ versus $1.456$). These experiments establish that the constrained geometric structure is a viable language model at this scale; the structure itself, and its synchronization reading, is the contribution. Ablations isolate the load-bearing components, and the result gives a compact bridge between self-attention and phase synchronization.

Related papers

• Time-Frequency Geometric Cross-Attention for Chunked Vision-Language-Action Models [10.591519354017803]
  We show how to repair two blind spots in a vision--action (VLA) policy.<n>We use a drop-in module to fine-tune a pretrained VLA.<n>We show how TFGCA improves in-distribution LIBERO by +1.5 on average, the OOD LIBERO-Plus by +6.3, the randomized average under RoboTwin domain randomization by +28.5, and the overall success rate on three real-robot AgiBot A2 tasks by +11.67 points.
  arXiv  Detail & Related papers  (2026-09-09T09:16:49Z)
• Model-level synthetic-flux control of hyperchaos order and matched-resource sensing in dissipative optomechanics [0.11545092788508222]
  We present a synthetic-flux phase $_rm syn$ control coordinate that selects the emphorder of a drive- and coupling-gated hyperchaos transition.<n>A phase-consistent Floquet--Lyapunov protocol (cross-checked by monodromy multipliers, dissipative volume balance $sum_i=nameTr(J)=-1.04$ and a 180-run three-seed audit) localizes the onset to a Neimark--Sacker bifurcation at $E*=num1.060
  arXiv  Detail & Related papers  (2026-09-02T17:13:43Z)
• Discrete Autoregressive Transformer for Generative Mechanism Synthesis [0.0]
  Planar path synthesis requires mechanisms whose coupler curves match a prescribed trajectory.<n>We report Chamfer distance and dynamic time warping after forward kinematics and geometric alignment.<n>We formulate synthesis as conditional autoregressive sequence modeling.
  arXiv  Detail & Related papers  (2026-06-16T01:46:21Z)
• Wavelet Variance Equipartition as a Threshold for World-Model Quality and Quantum Kernel TN-Simulability [0.0]
  We identify the wavelet scaling exponent $$ as a critical diagnostic.<n>We establish $= 1/2$ as a sharp transition boundary for the classical simulability of amplitude-encoded quantum kernels.<n>We prove this variance scales strictly as $Var[X] = (d-2)$.
  arXiv  Detail & Related papers  (2026-05-12T05:41:12Z)
• The Geometry of Forgetting: Temporal Knowledge Drift as an Independent Axis in LLM Representations [50.43168858368539]
  Large language models confidently produce outdated answers, and no existing method can detect them.<n>We show this is not an engineering failure but a structural one: temporal drift is encoded as a direction in the residual stream geometrically to both correctness and uncertainty.
  arXiv  Detail & Related papers  (2026-05-09T22:27:31Z)
• Mean-Pooled Cosine Similarity is Not Length-Invariant: Theory and Cross-Domain Evidence for a Length-Invariant Alternative [1.5718921092089344]
  Mean-pooled cosine similarity is the default metric for comparing neural representations across languages, modalities, and tasks.<n>Under the anisotropy that characterizes modern transformer representations, mean-pooled cosine grows monotonically in sequence length.<n>We argue that length-invariant metrics such as Centered Kernel Alignment should be the default for cross-representation comparisons.
  arXiv  Detail & Related papers  (2026-05-08T06:48:34Z)
• Coupled Query-Key Dynamics for Attention [6.775853253396773]
  We show evolving queries and keys emphjointly through shared learned dynamics before scoring.<n>On WikiText-103 at 60M parameters, coupled dynamics achieves 22.55--22.62 perplexity vs. 24.22 for standard attention.
  arXiv  Detail & Related papers  (2026-04-02T06:37:05Z)
• Token Sample Complexity of Attention [20.89022639697809]
  We estimate the rate at which attention computed on $n$ tokens converges to its infinite-token limit.<n>For compactly supported distributions, our first result shows that the attention map converges uniformly on a ball of radius $R$ at rate $C(R)/sqrtn$.<n>We also examine the regime in which the attention parameter tends to infinity and the softmax approaches a hardmax.
  arXiv  Detail & Related papers  (2025-12-11T14:02:34Z)
• Mesoscopic Fluctuations and Multifractality at and across Measurement-Induced Phase Transition [46.176861415532095]
  We explore statistical fluctuations over the ensemble of quantum trajectories in a model of two-dimensional free fermions.<n>Our results exhibit a remarkable analogy to Anderson localization, with $G_AB$ corresponding to two-terminal conductance.<n>Our findings lay the groundwork for mesoscopic theory of monitored systems, paving the way for various extensions.
  arXiv  Detail & Related papers  (2025-07-15T13:44:14Z)
• Unveiling Induction Heads: Provable Training Dynamics and Feature Learning in Transformers [54.20763128054692]
  We study how a two-attention-layer transformer is trained to perform ICL on $n$-gram Markov chain data. We prove that the gradient flow with respect to a cross-entropy ICL loss converges to a limiting model.
  arXiv  Detail & Related papers  (2024-09-09T18:10:26Z)
• Improved Algorithm for Adversarial Linear Mixture MDPs with Bandit Feedback and Unknown Transition [71.33787410075577]
  We study reinforcement learning with linear function approximation, unknown transition, and adversarial losses. We propose a new algorithm that attains an $widetildeO(dsqrtHS3K + sqrtHSAK)$ regret with high probability.
  arXiv  Detail & Related papers  (2024-03-07T15:03:50Z)
• Ancilla quantum measurements on interacting chains: Sensitivity of entanglement dynamics to the type and concentration of detectors [46.76612530830571]
  We consider a quantum many-body lattice system that is coupled to ancillary degrees of freedom (detectors'') We explore the dynamics of density and of entanglement entropy in the chain, for various values of $rho_a$ and $M$.
  arXiv  Detail & Related papers  (2023-11-21T21:41:11Z)
• Measurement-induced phase transition for free fermions above one dimension [46.176861415532095]
  Theory of the measurement-induced entanglement phase transition for free-fermion models in $d>1$ dimensions is developed. Critical point separates a gapless phase with $elld-1 ln ell$ scaling of the second cumulant of the particle number and of the entanglement entropy.
  arXiv  Detail & Related papers  (2023-09-21T18:11:04Z)
• Dynamical Signatures of Chaos to Integrability Crossover in $2\times 2$ Generalized Random Matrix Ensembles [0.0]
  We study the energy correlations by calculating the density and 2nd moment of the Nearest Neighbor Spacing (NNS) We observe that for large $N$ the 2nd moment of NNS and the relative depth of the correlation hole exhibit a second order phase transition at $gamma=2$.
  arXiv  Detail & Related papers  (2020-10-28T05:02:13Z)
• $O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers [71.31712741938837]
  We show that sparse Transformers with only $O(n)$ connections per attention layer can approximate the same function class as the dense model with $n2$ connections. We also present experiments comparing different patterns/levels of sparsity on standard NLP tasks.
  arXiv  Detail & Related papers  (2020-06-08T18:30:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.