Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention

• URL: http://arxiv.org/abs/2605.22791v1
• Date: Thu, 21 May 2026 17:44:57 GMT
• Title: Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention
• Abstract summary: Linear attention replaces the cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.<n>We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and Kimi Delta Attention.<n>Gated DeltaNet-2 achieves the strongest overall results among Mamba Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval.
• Score: 81.79922329750674
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Linear attention replaces the unbounded cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory. The hard part is not just what to forget, but how to edit this compressed memory without scrambling existing associations. Delta-rule models subtract the current read before writing a new value, and Kimi Delta Attention (KDA) sharpens forgetting with channel-wise decay. But the active edit still uses a single scalar gate to control two different things: how much old content to erase on the key side and how much new content to commit on the value side. We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and KDA by inheriting adaptive forgetting and channel-wise decay while addressing their shared limitation, the scalar tie between erasing and writing. Gated Delta Rule-2 separates these roles with a channel-wise erase gate b_t and a channel-wise write gate w_t, reducing to KDA when both gates collapse to the same scalar and to Gated DeltaNet when the decay also collapses. We derive a fast-weight update view, a chunkwise WY algorithm with channel-wise decay absorbed into asymmetric erase factors, and a gate-aware backward pass that preserves efficient parallel training. At 1.3B parameters trained on 100B FineWeb-Edu tokens, Gated DeltaNet-2 achieves the strongest overall results among Mamba-2, Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval. Its advantage is most pronounced on long-context RULER needle-in-a-haystack benchmarks, where it improves the evaluated multi-key retrieval setting and remains strong in both recurrent and hybrid settings. Code is available at https://github.com/NVlabs/GatedDeltaNet-2.

Related papers

• HEAT: Faster Fully Homomorphic Inference via Approximations-Weights Co-Adaptation [56.14416807766864]
  Homomorphic encryption allows a server to run a language model directly on encrypted user prompts.<n>Existing approaches fix the iteration counts uniformly across the model rather than tailoring them to each site's error tolerance.<n>We introduce Homomorphic Encryption-Aware Training (HEAT), a fine-tuning method that makes the per-nonline iteration counts learnable.
  arXiv  Detail & Related papers  (2026-09-01T18:02:02Z)
• The Query Knows What to Forget: A Second Erase Direction for Linear Attention [4.500637354443275]
  Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase vector from the key of the current token.<n>We introduce the Query-derived Erase Direction (QED)<n>It improves retrieval at every length past the training window, and it about doubles the usable context length on S-NIAH-1.
  arXiv  Detail & Related papers  (2026-08-13T18:04:53Z)
• Linear Attention Architectures: Mechanisms, Trade-offs, and Cross-Layer Routing [5.945255135393071]
  Self-attention lets each token retrieve information from the full context, but its cost in sequence length training and inference at long context is quadratic.<n>We express these mechanisms in a common recurrent-memory notation.<n>We introduce and evaluate lightweight cross-layer forwarding mechanisms for DeltaNet-style memories.
  arXiv  Detail & Related papers  (2026-07-08T22:14:14Z)
• Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences [51.38664601405696]
  We introduce preconditioned variants of DeltaNet, GDN, and KDA alongside efficient chunkwise parallel algorithms for computing them.<n>Our preconditioned delta-rule recurrences yield consistent performance improvements across synthetic recall benchmarks and language modeling at the 340M and 1B scale.
  arXiv  Detail & Related papers  (2026-04-22T21:38:25Z)
• FG$^2$-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control [31.409429284036776]
  Linear attention mechanisms have emerged as promising alternatives to softmax attention.<n>We introduce FG$2$-GDN, which replaces the scalar $_t$ with a channel-wise vector.<n>Experiments show that FG$2$-GDN and its variant improve associative recall and long-context understanding.
  arXiv  Detail & Related papers  (2026-04-21T03:15:28Z)
• Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression [53.48692193399171]
  Gated KalmaNet (GKA) is a layer that reduces the gap by accounting for the full past when predicting the next token.<n>We solve an online ridge regression problem at test time, with constant memory and linear compute cost in the sequence length.<n>On long-context, GKA excels at real-world RAG and LongQA tasks up to 128k tokens, achieving more than $10$% relative improvement over other fading memory baselines.
  arXiv  Detail & Related papers  (2025-11-26T03:26:37Z)
• Rectified Sparse Attention [61.7702154360081]
  Efficient long-sequence generation is a critical challenge for Large Language Models.<n>We propose Rectified Sparse Attention (ReSA), a simple yet effective method that combines block-sparse attention with periodic dense rectification.<n> Experiments across math reasoning, language modeling, and retrieval tasks demonstrate that ReSA achieves near-lossless generation quality.
  arXiv  Detail & Related papers  (2025-06-04T16:01:48Z)
• Gated Delta Networks: Improving Mamba2 with Delta Rule [64.58149707073915]
  Gated DeltaNet consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks.<n>We develop hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance.
  arXiv  Detail & Related papers  (2024-12-09T13:09:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.