Attention as Frustrated Synchronization

• URL: http://arxiv.org/abs/2606.18694v1
• Date: Wed, 17 Jun 2026 05:18:01 GMT
• Title: Attention as Frustrated Synchronization
• Abstract summary: We introduce the Frustrated Synchronization Network (FSN), whose token states are phases on a torus and whose entire value pathway is one learned complex coupling kernel over harmonics and a one-step delay.<n>The complex phases are static Kuramoto-Sakaguchi frustration angles, the signed harmonics are repulsive Daido components, and the delay term, which couples each token to the successors of the tokens it attends to, is algebraically identical to Kuramoto-Sakaguchi coupling.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A network of oscillators that synchronizes perfectly computes nothing further, so an attention architecture built from synchronization must locate its computation in structured departures from agreement. We introduce the Frustrated Synchronization Network (FSN), whose token states are phases on a torus and whose entire value pathway is one learned complex coupling kernel over harmonics and a one-step delay. Each component of the kernel is a frustration in the sense of the synchronization literature. The complex phases are static Kuramoto-Sakaguchi frustration angles, the signed harmonics are repulsive Daido components, and the delay term, which couples each token to the successors of the tokens it attends to, is algebraically identical to Kuramoto-Sakaguchi coupling whose frustration angle is the data's own transition, so next-token prediction is implemented as synchronization frustrated by the data. At matched one-million-parameter and training budgets on character-level text and code, the FSN's validation loss is below a tuned RoPE-SwiGLU transformer's at every epoch measured, and the comparison survives training the baseline to convergence: every thirty-epoch enwik8 seed finishes below the transformer's converged fifty-epoch loss of 1.611, and the FSN's completed fifty-epoch runs converge to 1.5953 +/- 0.0014. A variant with every feed-forward block replaced by mean-field coupling to learned collective modes, leaving no multilayer perceptron in the stack, tracks the transformer. On natural text the unfrustrated base layer falls behind the converged transformer at every copy depth, worst on long-range copy events; the kernel reverses the deficit at every depth of four and beyond. Headline comparisons are at the one-million-parameter scale; a scale ladder is complete through four million parameters with the advantage persisting, and remaining arms are marked as in progress.

Related papers

• From Concentration to Differentiation and Back: Routing Effective Rank in MoE Reasoning Cohorts [61.695980784491475]
  Test-time scaling produces cohorts of reasoning rollouts, yet there is no standard label-free account of how their internal computation reorganizes as inference unfolds.<n>We introduce routing effective rank deff, the entropy-effective dimensionality of a cross-rollout graph built from MoE expert-routing similarity.
  arXiv  Detail & Related papers  (2026-09-06T05:43:22Z)
• DFSC: Error-Controlled Differentiable Mittag-Leffler Propagation for Fractional Scientific Machine Learning [1.6010518212545237]
  PyTorch environment separates known fractional propagation from data-driven corrections.<n>DFSC supports dense, sparse, matrix-free, self-adjoint, generalized, and controlled complex operator paths.
  arXiv  Detail & Related papers  (2026-07-31T05:29:25Z)
• Stabilizing Extrapolation in Looped Transformers via Learned Stochastic Stopping [56.14767235650558]
  Looped Transformers, which repeatedly apply a shared transformer block, are an architecturally natural fit for variable-length computation tasks.<n>We trace this variance to the spurious correlation in simple algorithmic tasks between sequence length and number of loops.<n>Our work suggests that "when to stop" should be treated as a training design choice, not merely an inference-time-allocation rule.
  arXiv  Detail & Related papers  (2026-06-29T08:58:09Z)
• Discrete Autoregressive Transformer for Generative Mechanism Synthesis [0.0]
  Planar path synthesis requires mechanisms whose coupler curves match a prescribed trajectory.<n>We report Chamfer distance and dynamic time warping after forward kinematics and geometric alignment.<n>We formulate synthesis as conditional autoregressive sequence modeling.
  arXiv  Detail & Related papers  (2026-06-16T01:46:21Z)
• Kuramoto Attention: Synchronizing Self-Attention on the Torus [0.0]
  We introduce Kuramoto attention, a self-attention layer in which each hidden coordinate is an angle.<n>The layer scores tokens by gated cosine similarity, attends over previous phase states, and updates each token by the tangent component of the attention-weighted circular mean.<n>On enwiki8 character-level language modeling, the layer trains as a functional language model.
  arXiv  Detail & Related papers  (2026-06-10T02:24:04Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• N-vium: Mixture-of-Exits Transformer for Accelerated Exact Generation [68.47358899451255]
  N-vium is a mixture-of-exits transformer that partially parallelizes computation across depth on standard hardware.<n>N-vium attaches prediction heads at multiple depths and defines the next-token distribution as a learned mixture over these exits.
  arXiv  Detail & Related papers  (2026-05-13T08:46:17Z)
• Sub-Cubic Quantum Gate Synthesis via Stochastic Commutator Decomposition [0.0]
  Commutator Synthesis is a quantum gate compilation framework.<n>It integrates Kuperberg's sub-cubic Solovay-Kitaev exponent c near 1.44042 with the error-tailoring machinery of randomized compilation.
  arXiv  Detail & Related papers  (2026-05-03T20:41:21Z)
• The Recurrent Transformer: Greater Effective Depth and Efficient Decoding [48.9323408950142]
  We introduce the Recurrent Transformer, a simple architectural change where each layer attends to key-value pairs computed off its own activations.<n>We show that the architecture can emulate both (i) a conventional Transformer and (ii) token-to-gressive recurrent updates under mild assumptions.
  arXiv  Detail & Related papers  (2026-04-23T02:12:58Z)
• Early-Terminable Energy-Safe Iterative Coupling for Parallel Simulation of Port-Hamiltonian Systems [1.2091218906871861]
  This letter proposes a novel energy-safe, early-terminable iterative coupling for port-Hamiltonian subsystems.<n>We prove an augmented-storage inequality certifying discrete passivity of the coupled macro-step for any finite inner-iteration budget.<n>As the inner budget increases, the partitioned update converges to a monolithic discrete-time update.
  arXiv  Detail & Related papers  (2026-03-17T11:59:30Z)
• ATTS: Asynchronous Test-Time Scaling via Conformal Prediction [112.54016379556073]
  Large language models (LLMs) benefit from test-time scaling but are often hampered by high inference latency.<n>We introduce ATTS (Asynchronous Test-Time Scaling), a statistically guaranteed adaptive scaling framework.<n>We show that ATTS delivers up to 56.7x speedup in test-time scaling and a 4.14x throughput improvement.
  arXiv  Detail & Related papers  (2025-09-18T16:55:09Z)
• Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition [62.83832841523525]
  We propose a fast and accurate parallel transformer, termed Paraformer. It accurately predicts the number of output tokens and extract hidden variables. It can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup.
  arXiv  Detail & Related papers  (2022-06-16T17:24:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.