MDN: Parallelizing Stepwise Momentum for Delta Linear Attention

• URL: http://arxiv.org/abs/2605.05838v1
• Date: Thu, 07 May 2026 08:12:09 GMT
• Title: MDN: Parallelizing Stepwise Momentum for Delta Linear Attention
• Abstract summary: Linear Attention (LA) offers a promising paradigm for scaling large language models to long sequences.<n>Recent LA models such as Mamba2 and GDN interpret linear recurrences as closed-form online gradient descent.<n>Momentum DeltaNet (MDN) leverages Triton kernels to achieve comparable training throughput with competitive linear models.
• Score: 27.145174798663035
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Linear Attention (LA) offers a promising paradigm for scaling large language models (LLMs) to long sequences by avoiding the quadratic complexity of self-attention. Recent LA models such as Mamba2 and GDN interpret linear recurrences as closed-form online stochastic gradient descent (SGD), but naive SGD updates suffer from rapid information decay and suboptimal convergence in optimization. While momentum-based optimizers provide a natural remedy, they pose challenges in simultaneously achieving training efficiency and effectiveness. To address this, we develop a chunkwise parallel algorithm for LA with a stepwise momentum rule by geometrically reordering the update coefficients. Further, from a dynamical systems perspective, we analyze the momentum-based recurrence as a second-order system that introduces complex conjugate eigenvalues. This analysis guides the design of stable gating constraints. The resulting model, Momentum DeltaNet (MDN), leverages Triton kernels to achieve comparable training throughput with competitive linear models such as Mamba2 and KDA. Extensive experiments on the 400M and 1.3B parameter models demonstrate consistent performance improvements over strong baselines, including Transformers, Mamba2 and GDN, across diverse downstream evaluation benchmarks. Code: https://github.com/HuuYuLong/MomentumDeltaNet .

Related papers

• MambaADv2: Evolving Duality-enhanced State Space Model for Unsupervised Anomaly Detection [108.36437360254605]
  This paper proposes MambaADv2, a framework tailored for multi-class unsupervised anomaly detection.<n>MambaADv2 comprises a pre-trained encoder and a Mamba-inspired decoder, equipped with Duality-enhanced State Space (DSS) modules across multiple scales.<n>The structure of the Hybrid State Space (HSS) block is tailored by following the SSD-based Mamba lineage and incorporating Mamba3-style position-aware state-space modeling.
  arXiv  Detail & Related papers  (2026-06-22T10:14:06Z)
• Ada2MS: A Hybrid Optimization Algorithm Based on Exponential Mixing of Elementwise and Global Second-Moment Estimates [13.40796672049436]
  This paper proposes Ada2MS, an optimization algorithm that achieves a smooth transition between AdamW-like behavior and momentum-SGD-like behavior.<n>On the visual tasks evaluated in this study, Ada2MS obtains competitive results under a unified-comparison protocol.
  arXiv  Detail & Related papers  (2026-05-19T22:11:04Z)
• Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences [51.38664601405696]
  We introduce preconditioned variants of DeltaNet, GDN, and KDA alongside efficient chunkwise parallel algorithms for computing them.<n>Our preconditioned delta-rule recurrences yield consistent performance improvements across synthetic recall benchmarks and language modeling at the 340M and 1B scale.
  arXiv  Detail & Related papers  (2026-04-22T21:38:25Z)
• Mamba-3: Improved Sequence Modeling using State Space Principles [74.41028882099846]
  We introduce three core methodological improvements inspired by the state space model (SSM) viewpoint of linear models.<n>Together with architectural refinements, our Mamba-3 model achieves significant gains across retrieval, state-tracking, and downstream language modeling tasks.
  arXiv  Detail & Related papers  (2026-03-16T17:30:08Z)
• Adaptive Cubic Regularized Second-Order Latent Factor Analysis Model [14.755426957558868]
  High-dimensional and incompleteHDI datasets have become ubiquitous across various real-world applications.<n>We propose a two-fold approach to mitigate information instabilities.<n>The ACRS HDI demonstrate that the ALF represents higher representation than the faster advancing (SACR) models.
  arXiv  Detail & Related papers  (2025-07-03T03:15:54Z)
• Sequential-Parallel Duality in Prefix Scannable Models [68.39855814099997]
  Recent developments have given rise to various models, such as Gated Linear Attention (GLA) and Mamba.<n>This raises a natural question: can we characterize the full class of neural sequence models that support near-constant-time parallel evaluation and linear-time, constant-space sequential inference?
  arXiv  Detail & Related papers  (2025-06-12T17:32:02Z)
• Comba: Improving Bilinear RNNs with Closed-loop Control [57.800320390698516]
  We introduce the concept of Bilinear RNNs with a comprehensive analysis on the advantages and limitations of these models.<n>We propose a novel Bilinear RNN variant named Comba, which adopts a scalar-plus-low-rank state transition, with both state feedback and output feedback corrections.<n>We also implement a hardware-efficient chunk-wise parallel kernel in Triton and train models with 340M/1.3B parameters on large-scale corpus.
  arXiv  Detail & Related papers  (2025-06-03T05:44:50Z)
• LONGER: Scaling Up Long Sequence Modeling in Industrial Recommenders [23.70714095931094]
  Long-sequence optimized traNsformer for GPU-Efficient Recommenders.<n>Longer consistently outperforms strong baselines in offline metrics and online A/B testing.
  arXiv  Detail & Related papers  (2025-05-07T13:54:26Z)
• DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products [60.72655477351486]
  Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling.<n>Existing architectures face a fundamental trade-off between expressivity and efficiency, dictated by the structure of their state-transition matrices.
  arXiv  Detail & Related papers  (2025-02-14T16:59:05Z)
• Data-driven Nonlinear Parametric Model Order Reduction Framework using Deep Hierarchical Variational Autoencoder [5.521324490427243]
  Data-driven parametric model order reduction (MOR) method using a deep artificial neural network is proposed. LSH-VAE is capable of performing nonlinear MOR for the parametric of a nonlinear dynamic system with a significant number of degrees of freedom.
  arXiv  Detail & Related papers  (2023-07-10T02:44:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.