Kalman Delta Networks: Uncertainty-aware Associative Memory

• URL: http://arxiv.org/abs/2609.07816v1
• Date: Mon, 07 Sep 2026 17:53:29 GMT
• Title: Kalman Delta Networks: Uncertainty-aware Associative Memory
• Abstract summary: We reformulate recurrent associative memory as a linear--Gaussian state-space model.<n>Within KDNs, the transition propagates both the memory state and its uncertainty.<n>We show that KDN variants consistently improve perplexity and mean downstream accuracy over state-of-the-art linear-attention models.
• Score: 27.530297884143604
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Linear attention is increasingly used in frontier language models for efficient long-context inference and constant-memory decoding. Its fixed-size recurrent memory, however, requires an online decision at each token: what to write and how strongly to overwrite existing associations before knowing which information future queries will require. Delta-rule models learn this strength from the current token embedding but do not track confidence in the memory estimate, preventing each write from adapting to accumulated evidence. To represent this uncertainty explicitly, we reformulate recurrent associative memory as a linear--Gaussian state-space model, for which the Kalman filter is the optimal recursive estimator, and introduce a new family of models, Kalman Delta Networks (KDNs). Within KDNs, the transition propagates both the memory state and its uncertainty, allowing the Kalman gain to weight each residual write by accumulated evidence and observation reliability. Under this formulation, Delta-style updates emerge as a special case that substitutes a token-wise isotropic surrogate for predictive covariance and omits covariance tracking. Exact tracking, however, entails a dense, state-dependent Riccati recursion that is poorly suited to GPU-parallel linear-attention scans. To address this issue, we introduce two scan-compatible KDN approximations. Diagonal KDN projects each one-step posterior onto the diagonal Gaussian family through online mean-field variational inference, whereas Isotropic KDN uses an isotropic approximation with a single uncertainty scalar per head. Their uncertainty recurrences are Mobius maps, enabling associative scans with logarithmic parallel depth. Across controlled pretraining at 750M and 1.3B parameters, KDN variants consistently improve perplexity and mean downstream accuracy over state-of-the-art linear-attention models.

Related papers

• What Does a Discrete Diffusion Model Learn? [71.03603607324338]
  discrete diffusion model learns: a denoiser, a score ratio, or a bridge plug-in predictor.<n>We start with a rigorous derivation of the continuous-time Markov chain (CTMC) ELBO for any noising process.<n>Every identity is verified numerically, without approximation, on an exactly solvable model.
  arXiv  Detail & Related papers  (2026-07-06T17:56:11Z)
• Memory by Design: Probabilistic Sequence Layers [19.16406809753255]
  We introduce a way to derive efficient recurrent sequence maps from explicit assumptions about memory.<n>A design model writes evidence into memory by exact Bayesian filtering.<n>A query-dependent readout produces a predictive distribution whose mean is the layer output.
  arXiv  Detail & Related papers  (2026-05-29T11:13:00Z)
• OSDN: Improving Delta Rule with Provable Online Preconditioning in Linear Attention [12.93065958346192]
  Online Scaled DeltaNet improves JRT-style in-context recall by 32% over DeltaNet.<n> scaling to 1.3B parameters, it achieves a 39% reduction in the recall residual ratio.
  arXiv  Detail & Related papers  (2026-05-13T12:59:26Z)
• Kaczmarz Linear Attention [11.650692583508663]
  Linear recurrent models compress the context into a fixed-size state, making the rule that forgets, writes, and edits information a central design problem.<n>Gated DeltaNet (GDN) combines gated state decay with delta-rule residual writes, using a learnable coefficient to balance forgetting and update magnitude.<n>We propose Kaczmarz Linear (KLA), a one-scalar modification of GDN that preserves the state shape, gates, linear recurrence, and chunkwise parallel algorithm.
  arXiv  Detail & Related papers  (2026-05-09T01:07:01Z)
• Preconditioned DeltaNet: Curvature-aware Sequence Modeling for Linear Recurrences [51.38664601405696]
  We introduce preconditioned variants of DeltaNet, GDN, and KDA alongside efficient chunkwise parallel algorithms for computing them.<n>Our preconditioned delta-rule recurrences yield consistent performance improvements across synthetic recall benchmarks and language modeling at the 340M and 1B scale.
  arXiv  Detail & Related papers  (2026-04-22T21:38:25Z)
• Memory-Conditioned Flow-Matching for Stable Autoregressive PDE Rollouts [0.0]
  Autoregressive generative PDE solvers can be accurate one step ahead yet drift over long rollouts.<n>We show that eliminating unresolved variables yields an exact resolved evolution with a Markov term.<n>We then derive discrete Grnwall rollout bounds that separate memory approximation from conditional generation error.
  arXiv  Detail & Related papers  (2026-02-06T13:21:52Z)
• Variational Bayesian Flow Network for Graph Generation [54.94088904387278]
  We propose Variational Bayesian Flow Network (VBFN) for graph generation.<n>VBFN performs variational lifting to a tractable joint Gaussian variational belief family governed by structured precisions.<n>On synthetic and molecular graph datasets, VBFN improves fidelity and diversity, and surpasses baseline methods.
  arXiv  Detail & Related papers  (2026-01-30T03:59:38Z)
• Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression [53.48692193399171]
  Gated KalmaNet (GKA) is a layer that reduces the gap by accounting for the full past when predicting the next token.<n>We solve an online ridge regression problem at test time, with constant memory and linear compute cost in the sequence length.<n>On long-context, GKA excels at real-world RAG and LongQA tasks up to 128k tokens, achieving more than $10$% relative improvement over other fading memory baselines.
  arXiv  Detail & Related papers  (2025-11-26T03:26:37Z)
• Towards Continual Learning Desiderata via HSIC-Bottleneck Orthogonalization and Equiangular Embedding [55.107555305760954]
  We propose a conceptually simple yet effective method that attributes forgetting to layer-wise parameter overwriting and the resulting decision boundary distortion. Our method achieves competitive accuracy performance, even with absolute superiority of zero exemplar buffer and 1.02x the base model.
  arXiv  Detail & Related papers  (2024-01-17T09:01:29Z)
• Robust Implicit Networks via Non-Euclidean Contractions [63.91638306025768]
  Implicit neural networks show improved accuracy and significant reduction in memory consumption. They can suffer from ill-posedness and convergence instability. This paper provides a new framework to design well-posed and robust implicit neural networks.
  arXiv  Detail & Related papers  (2021-06-06T18:05:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.