Coupled Query-Key Dynamics for Attention

• URL: http://arxiv.org/abs/2604.01683v1
• Date: Thu, 02 Apr 2026 06:37:05 GMT
• Title: Coupled Query-Key Dynamics for Attention
• Abstract summary: We show evolving queries and keys emphjointly through shared learned dynamics before scoring.<n>On WikiText-103 at 60M parameters, coupled dynamics achieves 22.55--22.62 perplexity vs. 24.22 for standard attention.
• Score: 6.775853253396773
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Standard scaled dot-product attention computes scores from static, independent projections of the input. We show that evolving queries and keys \emph{jointly} through shared learned dynamics before scoring - which we call \textbf{coupled QK dynamics} - improves language modeling perplexity and training stability. On WikiText-103 at 60M parameters, coupled dynamics achieves 22.55--22.62 perplexity vs.\ 24.22 for standard attention ($-$6.6--6.9\%), with only 0.11\% additional parameters (shared across both instantiations). A structural ablation isolates coupling as the active ingredient: a symplectic (Hamiltonian) and a non-symplectic (Euler) integrator perform identically when both couple Q and K, while an uncoupled MLP baseline of matched capacity reaches only 23.81 with 8$\times$ higher seed variance. The integration step count (1--7) is similarly irrelevant - a single coupled step suffices. A compute-matched comparison reveals that coupling is a \emph{sample-efficiency} mechanism: standard attention trained for 2.4$\times$ longer (matching wall-clock) reaches the same perplexity, but requires 2.4$\times$ more tokens. The advantage scales to 150M ($-$6.7\%) but narrows at 350M ($-$1.0\%), where Differential Attention (18.93) overtakes coupled dynamics (19.35). The benefit is corpus-dependent: coupling helps on domain-coherent text (WikiText-103 $-$6.6\%, PubMed $-$4.5\%) but degrades on heterogeneous web text ($+$10.3\%) and shows no benefit on GLUE. We characterize when coupling helps and when it does not, providing practical guidelines.

Related papers

• Steering Under Compression: Dose-Response, Capability Cost, and Failure Asymmetry in Quantized LLMs [0.0]
  Inference-time steering enables behavioral control of large language models without parameter modification.<n>Post-training quantization reduces memory and compute costs for deployment.<n>We systematically study activation steering under weight-only quantization.
  arXiv  Detail & Related papers  (2026-09-06T08:41:42Z)
• Emotion as a Distribution: Joint Valence-Arousal Probability Learning for Speaker-Independent Multimodal Emotion Recognition [0.4779196219827507]
  multimodal recognizers collapse human emotion onto a single hard label.<n>We argue the recognizer should instead expose a distribution over affective space.<n>Our system emits a $9times9$ probability matrix over the Valence-Arousal plane.
  arXiv  Detail & Related papers  (2026-09-04T22:34:14Z)
• Ockhamareto: Pareto-Gated Segment-Level Credit Assignment for Concise Unit-Test Generation with Reinforcement Learning [64.49364696628294]
  We introduce textbfOckhamareto, a single-shot GRPO framework for unit-test generation and selection.<n>Ockhamareto has two principal components: (i)a emphPareto-gated Bonus that rewards only rollouts non-dominated in(mutation, $-$#tests) space, and (ii)emphToken-level Segment Credit, which attributes each test's marginal mutation kills back to the tokens of its unit-test block.
  arXiv  Detail & Related papers  (2026-08-25T12:20:02Z)
• IndicQE-APE: A Benchmark for Quality Estimation and Automatic Post-Editing for Indic Languages [77.88427845181154]
  We consolidate the WMT 2020--2024 shared-task lineage with an extended English--Malayalam resource into indicqe: $126,754$ instances over nine directional pairs.<n>We benchmark six prompted LLMs and three COMET metrics on segment-level QE, and three systems on APE.
  arXiv  Detail & Related papers  (2026-08-17T09:50:52Z)
• Kernelized Linear Attention: Breaking the Capacity Wall with Symmetric Cones [4.847095054351615]
  Linear attention promises constant-time recurrent inference but degrades sharply on associative recall.<n>We introduce Kernelized Linear Attention Activations (KATA), a framework whose feature maps are derived from first principles.<n>We show that rank-one positive semi-definite features offer a favorable capacity--interference tradeoff.
  arXiv  Detail & Related papers  (2026-07-19T21:46:09Z)
• Automated Proving of Shannon-Type Entropy Inequalities via Fine-Tuned Language Models and Guided Tree Search [50.16356451328644]
  Proving Shannon-type entropy inequalities is a fundamental task in information theory.<n>We investigate whether small-scale large language models, fine-tuned on atomic proof steps, can automate this process.<n>We show that GPT-5.5 solves 1.7% samples under zero-shot prompting while Psitip solves 33.3% samples.
  arXiv  Detail & Related papers  (2026-06-04T05:43:12Z)
• Benchmarking Speech-to-Speech Translation Models [55.00303727199927]
  Speech-to-speech translation (S2ST) has advanced rapidly, but offline evaluation lacks a unified protocol.<n>We introduce a benchmarking framework integrating 46 metrics across eight dimensions.<n>We deploy it on 1,248 model-language configurations from FLEURS and CVSS.
  arXiv  Detail & Related papers  (2026-06-02T07:01:33Z)
• Models Can Model, But Can't Bind: Structured Grounding in Text-to-Optimization [54.749573452394664]
  We find that accuracy collapses as instance data grows, even when the formulation itself is simple.<n>We address this via a simple inference-time approach, BIND, which externalizes numeric data to structured files so the model binds data rather than from prompt prompt.<n>We validate our hypothesis by finetuning a model exclusively on binding and show that it outperforms end-to-end SFT and RL across three structurally distinct optimization categories.
  arXiv  Detail & Related papers  (2026-05-20T21:25:41Z)
• The Structural Attention Tax: How Retrieval Format Hijacks In-Context Learning Independent of Content [11.226669792850645]
  We formalise a phenomenon we term the structural attention tax: knowledge graph (KG) triples.<n>KG triples capture 2-3x more attention per token than semantically equivalent natural-language text.<n>We develop a formal framework decomposing attention scores into semantic and structural components.
  arXiv  Detail & Related papers  (2026-04-21T08:47:00Z)
• Equifinality in Mixture of Experts: Routing Topology Does Not Determine Language Modeling Quality [0.0]
  We build a sparse Mixture-of-Experts (MoE) architecture using cosine-similarity routing against learned centroids.<n>We find that routing topology does not determine perplexity (PPL)<n>Expert-level and causal controllability -- which coexist with topology-level equifinality -- are explored in a companion paper.
  arXiv  Detail & Related papers  (2026-04-15T21:02:02Z)
• Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents [0.25891009512378166]
  Chain-of-thought (CoT) reasoning is widely assumed to improve agent performance.<n>We present a systematic study of CoT budget effects on function-calling agents.<n>Our central finding is a striking non-monotonic pattern on Qwen2.5-1.5B-Instruct: brief reasoning (32 tokens) dramatically improves accuracy by 45% relative over direct answers.
  arXiv  Detail & Related papers  (2026-04-02T15:25:13Z)
• Scaling Attention via Feature Sparsity [50.64995497733461]
  Scaling Transformers to ultra-long contexts is bottlenecked by the $O(n2 d)$ cost of self-attention.<n>We propose Sparse Feature Attention, where queries and keys are represented as $k$-sparse codes that preserve high-dimensional expressivity.<n>Across GPT-2 and Qwen3 pretraining, SFA matches dense baselines while improving speed by up to $2.5times$ and reducing FLOPs and KV-cache by nearly 50%.
  arXiv  Detail & Related papers  (2026-03-17T08:41:50Z)
• Thin Keys, Full Values: Reducing KV Cache via Low-Dimensional Attention Selection [15.230462656535034]
  Standard transformer attention uses identical dimensionality for queries, keys, and values ($d_q = d_k = d_v = dmodel$)<n>Our insight is that these components serve fundamentally different roles, and this symmetry is unnecessary.<n>We argue that selection is an inherently lower-dimensional operation than value transfer, requiring only $BigO(log N) dimensions to distinguish among $N relevant patterns.
  arXiv  Detail & Related papers  (2026-02-16T23:45:39Z)
• Decoherence, Perturbations and Symmetry in Lindblad Dynamics [0.0]
  We extend a perturbative Dyson-type treatment and discrete-symmetry constraints from the Schrdinger and von Neumann equations to a dephasing Lindblad framework.<n>This work develops further the odd-symmetric formulation -- based on realism and dual temporal boundary conditions -- from general dynamical considerations to specific tools of quantum mechanics.
  arXiv  Detail & Related papers  (2026-02-14T23:23:29Z)
• Balancing Centralized Learning and Distributed Self-Organization: A Hybrid Model for Embodied Morphogenesis [0.0]
  We investigate how to couple a learnable brain-like'' controller to a cell-like'' Gray--Scott substrate to steer pattern formation with minimal effort.<n>A compact convolutional policy is embedded in a differentiable PyTorch reaction--diffusion simulator.
  arXiv  Detail & Related papers  (2025-11-13T09:05:27Z)
• Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining [36.70261826975876]
  We present Multipole Semantic Attention (MuSe), an efficient approximation of softmax attention.<n>Our method addresses the quadratic computational complexity of transformers in the context length by clustering queries and keys separately.<n>For causal attention, we develop a hierarchical block decomposition that combines exact local computation with efficient long-range approximation.
  arXiv  Detail & Related papers  (2025-09-12T16:58:17Z)
• Near-Optimal Clustering in Mixture of Markov Chains [74.3828414695655]
  We study the problem of clustering $T$ trajectories of length $H$, each generated by one of $K$ unknown ergodic Markov chains over a finite state space of size $S$.<n>We derive an instance-dependent, high-probability lower bound on the clustering error rate, governed by the weighted KL divergence between the transition kernels of the chains.<n>We then present a novel two-stage clustering algorithm.
  arXiv  Detail & Related papers  (2025-06-02T05:10:40Z)
• Learning Adaptive Parallel Reasoning with Language Models [70.1745752819628]
  We propose Adaptive Parallel Reasoning (APR), a novel reasoning framework that enables language models to orchestrate both serialized and parallel computations end-to-end.<n> APR generalizes existing reasoning methods by enabling adaptive multi-threaded inference using spawn() and join() operations.<n>A key innovation is our end-to-end reinforcement learning strategy, optimizing both parent and child inference threads to enhance task success rate without requiring predefined reasoning structures.
  arXiv  Detail & Related papers  (2025-04-21T22:29:02Z)
• Parameterization of Cross-Token Relations with Relative Positional Encoding for Vision MLP [52.25478388220691]
  Vision multi-layer perceptrons (MLPs) have shown promising performance in computer vision tasks. They use token-mixing layers to capture cross-token interactions, as opposed to the multi-head self-attention mechanism used by Transformers. We propose a new positional spacial gating unit (PoSGU) to efficiently encode the cross-token relations for token mixing.
  arXiv  Detail & Related papers  (2022-07-15T04:18:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.