OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention

• URL: http://arxiv.org/abs/2605.13473v1
• Date: Wed, 13 May 2026 12:59:26 GMT
• Title: OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention
• Abstract summary: Online Scaled DeltaNet improves JRT-style in-context recall by 32% over DeltaNet.<n> scaling to 1.3B parameters, it achieves a 39% reduction in the recall residual ratio.
• Score: 12.93065958346192
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Linear attention and state-space models offer constant-memory alternatives to softmax attention, but often struggle with in-context associative recall. The Delta Rule mitigates this by writing each token via one step of online gradient descent. However, its step size relies on a single scalar gate that ignores the feature-wise curvature of the inner objective. We propose Online Scaled DeltaNet (OSDN), which augments the scalar gate with a diagonal preconditioner updated online via hypergradient feedback. Crucially, this right-preconditioning is algebraically equivalent to a per-feature scaling of the write-side key. This equivalence allows OSDN to strictly preserve the hardware-friendly chunkwise parallel pipeline of DeltaNet without incurring high-dimensional state overhead. Theoretically, by exploiting the exact-quadratic structure of the inner regression loss, we establish super-geometric convergence against a right-Newton comparator and prove an algorithm-aligned token-local residual contraction bound. To handle non-stationary contexts, we further introduce Adaptive Preconditioner Forgetting (APF) to dynamically refresh stale calibration. Empirically, OSDN demonstrates strong performance across scales. At the 340M-parameter scale, OSDN improves JRT-style in-context recall by 32% over DeltaNet. Scaling to 1.3B parameters, it achieves a 39% reduction in the recall residual ratio while maintaining parity on general downstream tasks (e.g., perplexity and LongBench) -- demonstrating that our online-preconditioning mechanism effectively transfers and amplifies at the billion-parameter scale.

Related papers

• Kalman Delta Networks: Uncertainty-aware Associative Memory [27.530297884143604]
  We reformulate recurrent associative memory as a linear--Gaussian state-space model.<n>Within KDNs, the transition propagates both the memory state and its uncertainty.<n>We show that KDN variants consistently improve perplexity and mean downstream accuracy over state-of-the-art linear-attention models.
  arXiv  Detail & Related papers  (2026-09-07T17:53:29Z)
• Edge-aware Decoding for Neural Asymmetric Routing [11.712945127862204]
  We propose a decoder-design principle for neural asymmetric routing.<n>The final score should explicitly expose transition-level quantities suggested by the problem's cost-to-go structure.<n>We instantiate this principle with an edge-aware decoder that adds candidate-specific terms for the current directed edge.
  arXiv  Detail & Related papers  (2026-06-01T12:03:59Z)
• Latent Recurrent Transformer: Architecture Exploration, Training Strategies, and Scaling Behavior [107.2098567818173]
  Latent Recurrent Transformer (LRT) is a lightweight augmentation of autoregressive transformers.<n>LRT reuses a high-level source-layer hidden state from the previous token as recurrent memory for the next token.
  arXiv  Detail & Related papers  (2026-05-26T10:10:26Z)
• Kaczmarz Linear Attention [11.650692583508663]
  Linear recurrent models compress the context into a fixed-size state, making the rule that forgets, writes, and edits information a central design problem.<n>Gated DeltaNet (GDN) combines gated state decay with delta-rule residual writes, using a learnable coefficient to balance forgetting and update magnitude.<n>We propose Kaczmarz Linear (KLA), a one-scalar modification of GDN that preserves the state shape, gates, linear recurrence, and chunkwise parallel algorithm.
  arXiv  Detail & Related papers  (2026-05-09T01:07:01Z)
• Channel-Level Relation to Attentive Aggregation with Neighborhood-Homogeneity Constraint for Point Cloud Analysis [14.63838637349137]
  In 3D point cloud understanding, the core challenge lies in accurately capturing discriminative features within complex neighborhoods.<n>Existing methods explore feature correlation discrimination but are limited to point-level spatial distribution or channel responses.<n>We propose PointCRA, a novel network with a channel-level metric-based enhancement mechanism.
  arXiv  Detail & Related papers  (2026-05-04T08:59:36Z)
• Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences [51.38664601405696]
  We introduce preconditioned variants of DeltaNet, GDN, and KDA alongside efficient chunkwise parallel algorithms for computing them.<n>Our preconditioned delta-rule recurrences yield consistent performance improvements across synthetic recall benchmarks and language modeling at the 340M and 1B scale.
  arXiv  Detail & Related papers  (2026-04-22T21:38:25Z)
• BASIS: Balanced Activation Sketching with Invariant Scalars for "Ghost Backpropagation" [0.0]
  Activation memory required for exact backpropagation scales linearly with network depth, context length, and feature dimensionality.<n>In this paper, we introduce an efficient backpropagation algorithm that fully decouples activation memory from the batch and sequence dimensions.
  arXiv  Detail & Related papers  (2026-03-05T20:38:25Z)
• Deep Delta Learning [91.75868893250662]
  We introduce Deep Delta Learning (DDL), a novel architecture that generalizes the standard residual connection.<n>We provide a spectral analysis of this operator, demonstrating that the gate $(mathbfX)$ enables dynamic between identity mapping, projection, and geometric reflection.<n>This unification empowers the network to explicitly control the spectrum of its layer-wise transition operator, enabling the modeling of complex, non-monotonic dynamics.
  arXiv  Detail & Related papers  (2026-01-01T18:11:38Z)
• Parallel Decoder Transformer: Model-Internal Parallel Decoding with Speculative Invariance via Note Conditioning [0.0]
  We introduce the textbfParallel Decoder Transformer (PDT), a parameter-efficient architecture that embeds coordination primitives directly into the inference process of a frozen pre-trained model.<n>PDT achieves effective self-correction, reaching textbf77.8% precision in coverage prediction and recovering approximate serial semantics without modifying the trunk weights.
  arXiv  Detail & Related papers  (2025-12-10T20:19:10Z)
• Comba: Improving Bilinear RNNs with Closed-loop Control [57.800320390698516]
  We introduce the concept of Bilinear RNNs with a comprehensive analysis on the advantages and limitations of these models.<n>We propose a novel Bilinear RNN variant named Comba, which adopts a scalar-plus-low-rank state transition, with both state feedback and output feedback corrections.<n>We also implement a hardware-efficient chunk-wise parallel kernel in Triton and train models with 340M/1.3B parameters on large-scale corpus.
  arXiv  Detail & Related papers  (2025-06-03T05:44:50Z)
• DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Products [60.72655477351486]
  Linear Recurrent Neural Networks (linear RNNs) have emerged as competitive alternatives to Transformers for sequence modeling.<n>Existing architectures face a fundamental trade-off between expressivity and efficiency, dictated by the structure of their state-transition matrices.
  arXiv  Detail & Related papers  (2025-02-14T16:59:05Z)
• Dynamic Convolution for 3D Point Cloud Instance Segmentation [146.7971476424351]
  We propose an approach to instance segmentation from 3D point clouds based on dynamic convolution. We gather homogeneous points that have identical semantic categories and close votes for the geometric centroids. The proposed approach is proposal-free, and instead exploits a convolution process that adapts to the spatial and semantic characteristics of each instance.
  arXiv  Detail & Related papers  (2021-07-18T09:05:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.