Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing

• URL: http://arxiv.org/abs/2607.07953v1
• Date: Wed, 08 Jul 2026 22:14:14 GMT
• Title: Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing
• Abstract summary: Self-attention lets each token retrieve information from the full context, but its cost in sequence length training and inference at long context is quadratic.<n>We express these mechanisms in a common recurrent-memory notation.<n>We introduce and evaluate lightweight cross-layer forwarding mechanisms for DeltaNet-style memories.
• Score: 5.945255135393071
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-attention lets each token retrieve information from the full context, but its quadratic cost in sequence length limits training and inference at long context. This paper presents a comparative study of softmax attention and four recent recurrent linear-attention architectures: DeltaNet, Gated DeltaNet, Kimi Delta Attention, and Gated DeltaNet-2. We express these mechanisms in a common recurrent-memory notation, making explicit how they differ in expressivity, memory decay, erase and write control, training throughput, and implementation complexity. Our experiments center on 350M-parameter models trained for 15B tokens, and include optimizer and learning-rate comparisons, hybrid-versus-pure stack comparisons, sequence-length runtime measurements, larger DeltaNet runs at 1.3B and 3B parameters, and a small set of downstream evaluations. The reported speed results measure training throughput and iteration time; we do not provide an empirical inference-speed benchmark. Within the reported 350M-parameter, 15B-token sweep, Kimi Delta Attention with Muon reaches the lowest final validation loss, a pure Gated DeltaNet stack trained with AdamW has the highest normalized training throughput, hybrid stacks generally improve loss at a throughput cost, and Muon consistently lowers final validation loss relative to AdamW in the matched architecture settings we evaluate. We introduce and evaluate lightweight cross-layer routing mechanisms for DeltaNet-style memories. The most natural DeltaNet-inspired formulation, forwarding a lower layer's delta-rule write error into the next layer's value target, does not improve over matched baselines. Routing into the aligned hidden stream and forwarding the write value instead yields a modest improvement in the matched runs we report: Cross-Layer Value Routing (CLVR) lowers final validation loss for both DeltaNet and Gated DeltaNet.

Related papers

• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• Autoencoder Architectures for Athlete Performance Scoring from Wearable Telemetry [0.0]
  This paper evaluates five dimensionality reduction models, three autoencoder variants, PCA, and a Variational Autoencoder, on their ability to compress nine sensor runner profiles into a single scalar performance indicator, the latent score.
  arXiv  Detail & Related papers  (2026-06-26T14:41:24Z)
• REVES: REvision and VErification--Augmented Training for Test-Time Scaling [53.197756110943395]
  We propose a two-stage iterative framework that alternates between online data/prompt augmentation and policy optimization.<n>We observe gains of +6.5 points over the RL baseline and +4.0 points over standard multi-turn training.
  arXiv  Detail & Related papers  (2026-06-17T10:37:23Z)
• Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention [81.79922329750674]
  Linear attention replaces the cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.<n>We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and Kimi Delta Attention.<n>Gated DeltaNet-2 achieves the strongest overall results among Mamba Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval.
  arXiv  Detail & Related papers  (2026-05-21T17:44:57Z)
• Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences [51.38664601405696]
  We introduce preconditioned variants of DeltaNet, GDN, and KDA alongside efficient chunkwise parallel algorithms for computing them.<n>Our preconditioned delta-rule recurrences yield consistent performance improvements across synthetic recall benchmarks and language modeling at the 340M and 1B scale.
  arXiv  Detail & Related papers  (2026-04-22T21:38:25Z)
• The Delta Learning Hypothesis: Preference Tuning on Weak Data can Yield Strong Gains [50.66245575710432]
  We show that paired preference data consisting of individually weak data points can enable gains beyond the strength of each individual data point.<n>Our work shows that models can learn surprisingly well from paired data that might typically be considered weak.
  arXiv  Detail & Related papers  (2025-07-08T17:14:44Z)
• DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products [60.72655477351486]
  Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling.<n>Existing architectures face a fundamental trade-off between expressivity and efficiency, dictated by the structure of their state-transition matrices.
  arXiv  Detail & Related papers  (2025-02-14T16:59:05Z)
• Gated Delta Networks: Improving Mamba2 with Delta Rule [64.58149707073915]
  Gated DeltaNet consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks.<n>We develop hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance.
  arXiv  Detail & Related papers  (2024-12-09T13:09:04Z)
• Freshness or Accuracy, Why Not Both? Addressing Delayed Feedback via Dynamic Graph Neural Networks [23.952923773407043]
  Delayed feedback problem is one of the most pressing challenges in predicting conversion rate. We propose Delayed Feedback Modeling by Dynamic Graph Neural Network (DGDFEM) It includes three stages, i.e., preparing a data pipeline, building a dynamic graph, and training a CVR prediction model.
  arXiv  Detail & Related papers  (2023-08-15T23:49:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.