FG$^2$-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control

• URL: http://arxiv.org/abs/2604.19021v1
• Date: Tue, 21 Apr 2026 03:15:28 GMT
• Title: FG$^2$-GDN: Enhancing Long-Context Gated Delta Networks with Doubly Fine-Grained Control
• Abstract summary: Linear attention mechanisms have emerged as promising alternatives to softmax attention.<n>We introduce FG$2$-GDN, which replaces the scalar $_t$ with a channel-wise vector.<n>Experiments show that FG$2$-GDN and its variant improve associative recall and long-context understanding.
• Score: 31.409429284036776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Linear attention mechanisms have emerged as promising alternatives to softmax attention, offering linear-time complexity during inference. Recent advances such as Gated DeltaNet (GDN) and Kimi Delta Attention (KDA) have demonstrated that the delta rule, an online gradient descent update, enables superior associative recall compared to simple additive updates. While KDA refined the coarse head-wise decay gate into channel-wise decay, the learning rate $β_t$ in the delta update remains a scalar, limiting the model's capacity for dimension-specific adaptation. We introduce FG$^2$-GDN, which replaces the scalar $β_t$ with a channel-wise vector analogous to the transition from SGD to per-coordinate adaptive optimizers such as AdaGrad and Adam. We further propose FG$^2$-GDN+, which decouples the scaling for keys and values, enabling independent control of erasure strength and write strength. Experiments on synthetic and real-world benchmarks show that FG$^2$-GDN and its variant improve associative recall and long-context understanding over GDN and KDA, with comparable computational efficiency.

Related papers

• Kalman Delta Networks: Uncertainty-aware Associative Memory [27.530297884143604]
  We reformulate recurrent associative memory as a linear--Gaussian state-space model.<n>Within KDNs, the transition propagates both the memory state and its uncertainty.<n>We show that KDN variants consistently improve perplexity and mean downstream accuracy over state-of-the-art linear-attention models.
  arXiv  Detail & Related papers  (2026-09-07T17:53:29Z)
• Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention [81.79922329750674]
  Linear attention replaces the cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.<n>We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and Kimi Delta Attention.<n>Gated DeltaNet-2 achieves the strongest overall results among Mamba Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval.
  arXiv  Detail & Related papers  (2026-05-21T17:44:57Z)
• OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention [12.93065958346192]
  Online Scaled DeltaNet improves JRT-style in-context recall by 32% over DeltaNet.<n> scaling to 1.3B parameters, it achieves a 39% reduction in the recall residual ratio.
  arXiv  Detail & Related papers  (2026-05-13T12:59:26Z)
• Kaczmarz Linear Attention [11.650692583508663]
  Linear recurrent models compress the context into a fixed-size state, making the rule that forgets, writes, and edits information a central design problem.<n>Gated DeltaNet (GDN) combines gated state decay with delta-rule residual writes, using a learnable coefficient to balance forgetting and update magnitude.<n>We propose Kaczmarz Linear (KLA), a one-scalar modification of GDN that preserves the state shape, gates, linear recurrence, and chunkwise parallel algorithm.
  arXiv  Detail & Related papers  (2026-05-09T01:07:01Z)
• MDN: Parallelizing Stepwise Momentum for Delta Linear Attention [27.145174798663035]
  Linear Attention (LA) offers a promising paradigm for scaling large language models to long sequences.<n>Recent LA models such as Mamba2 and GDN interpret linear recurrences as closed-form online gradient descent.<n>Momentum DeltaNet (MDN) leverages Triton kernels to achieve comparable training throughput with competitive linear models.
  arXiv  Detail & Related papers  (2026-05-07T08:12:09Z)
• Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences [51.38664601405696]
  We introduce preconditioned variants of DeltaNet, GDN, and KDA alongside efficient chunkwise parallel algorithms for computing them.<n>Our preconditioned delta-rule recurrences yield consistent performance improvements across synthetic recall benchmarks and language modeling at the 340M and 1B scale.
  arXiv  Detail & Related papers  (2026-04-22T21:38:25Z)
• DeltaSeg: Tiered Attention and Deep Delta Learning for Multi-Class Structural Defect Segmentation [1.2904450317782683]
  This paper presents a U-shaped encoder-decoder architecture with a tiered attention strategy that integrates Squeeze-and-Excitation (SE) channel attention in the encoder, Coordinate Attention at the bottleneck and decoder, and a novel Deep Delta Attention (DDA) mechanism in the skip connections.<n>Across both benchmarks, DeltaSeg consistently outperforms 12 competing architectures including U-Net, SA-UNet, UNet3+, SegFormer, Swin-UNet, EGE-UNet, FPN, and Mobile-UNETR.
  arXiv  Detail & Related papers  (2026-04-20T18:49:16Z)
• DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products [60.72655477351486]
  Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling.<n>Existing architectures face a fundamental trade-off between expressivity and efficiency, dictated by the structure of their state-transition matrices.
  arXiv  Detail & Related papers  (2025-02-14T16:59:05Z)
• Gated Delta Networks: Improving Mamba2 with Delta Rule [64.58149707073915]
  Gated DeltaNet consistently surpasses existing models like Mamba2 and DeltaNet across multiple benchmarks.<n>We develop hybrid architectures that combine Gated DeltaNet layers with sliding window attention or Mamba2 layers, achieving both improved training efficiency and superior task performance.
  arXiv  Detail & Related papers  (2024-12-09T13:09:04Z)
• LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inference [19.5669231249754]
  We present LinGCN, a framework designed to reduce multiplication depth and optimize the performance of HE based GCN inference. Remarkably, LinGCN achieves a 14.2x latency speedup relative to CryptoGCN, while preserving an inference accuracy of 75% and notably reducing multiplication depth.
  arXiv  Detail & Related papers  (2023-09-25T17:56:54Z)
• Switch and Conquer: Efficient Algorithms By Switching Stochastic Gradient Oracles For Decentralized Saddle Point Problems [1.2277343096128712]
  We develop an inexact primal hybrid gradient (inexact PDHG) procedure that allows generic gradient oracles to update the primal and dual variables. We show that utilizing the best convergence phases of GSG and SVRG oracles makes C-DPSSG well suited for obtaining solutions of low/medium accuracy faster.
  arXiv  Detail & Related papers  (2023-09-02T17:48:42Z)
• DiNADO: Norm-Disentangled Neurally-Decomposed Oracles for Controlling Language Models [55.06442277395388]
  NeurAlly-Decomposed Oracle (NADO) is a powerful approach for controllable generation with large language models. Vanilla NADO suffers from vanishing gradient for low-probability control signals. We propose a improved version of the NADO algorithm, namely DiNADO, which improves the performance of the NADO algorithm.
  arXiv  Detail & Related papers  (2023-06-20T18:36:52Z)
• On the Global Convergence of Training Deep Linear ResNets [104.76256863926629]
  We study the convergence of gradient descent (GD) and gradient descent (SGD) for training $L$-hidden-layer linear residual networks (ResNets) We prove that for training deep residual networks with certain linear transformations at input and output layers, both GD and SGD can converge to the global minimum of the training loss.
  arXiv  Detail & Related papers  (2020-03-02T18:34:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.