Daily Paper Briefing

Abstract Overview

This paper analyzes the expressivity limits of Kimi Delta Attention (KDA) and introduces Complex KDA (CKDA), a variant that extends the gate range to [-1,1] and the delta-rule coefficient to [0,2]. The core theoretical claim is that these two extensions let a single diagonal-plus-rank-one recurrent transition realize planar rotations, including complex-conjugate eigenvalue pairs, while remaining non-expansive and efficient. The authors prove that every orthogonal diagonal-plus-rank-one matrix can be represented as a CKDA transition and derive state-tracking expressivity results for finite groups, regular languages, and weighted finite automata under specified assumptions. Experiments evaluate state-tracking tasks, periodic waveform continuation, implementation throughput, and language modeling, relating the observed behavior back to the proposed rotational mechanism.

Novelty

The main novelty is showing that KDA's channel-wise gate can supply a second reflection, so that with signed gates and an extended beta range a single KDA-style update can implement 2D rotations without composing multiple delta-rule transitions. The work also provides a characterization of CKDA transitions by proving that all orthogonal diagonal-plus-rank-one matrices are exactly realizable in this form.

Results

The paper proves that CKDA matches the state-tracking expressivity of DeltaProduct-2 in several settings while preserving diagonal-plus-rank-one, non-expansive transitions, and that three CKDA layers suffice for arbitrary finite-group word problems, with stronger weighted-automata results when beta > 2 is allowed. Empirically, CKDA gives the strongest length extrapolation among the tested KDA range settings on S3, S4, and periodic waveform continuation, retains about 96-97% of standard KDA throughput, and in 1.3B-parameter language modeling performs similarly to a KDA baseline while outperforming Transformer and several linear-RNN baselines reported in the study.

Key Points

1. CKDA extends KDA with signed channel-wise gates and beta in [0,2], enabling single-step planar rotations and at most one complex-conjugate eigenvalue pair in a non-expansive transition.
2. The theory characterizes CKDA's transition family, proves exact realization of orthogonal diagonal-plus-rank-one matrices, and establishes finite-group and multilayer state-tracking expressivity bounds together with a one-layer limitation for S5 under stated assumptions.
3. Experiments link the theory to practice: learned CKDA heads develop near-sign gates, beta values near 2, and complex spectra; CKDA also improves long-range extrapolation on symbolic and periodic tasks while maintaining near-baseline KDA efficiency.

References

• arXiv: https://arxiv.org/abs/2609.24797v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2609.24797v1
• Hugging Face Papers: https://huggingface.co/papers/2609.24797