Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity

• URL: http://arxiv.org/abs/2607.07386v1
• Date: Wed, 08 Jul 2026 13:17:19 GMT
• Title: Sparse Delta Memory: Scaling the State of Linear RNNs through Sparsity
• Abstract summary: We introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme.<n>We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks.
• Score: 34.491686452909526
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Linear attention models allow a fixed state size and a fixed amount of compute per token. However, due to their limited state size, linear attention models fall behind in long-context recall compared to softmax-attention-based transformer architectures. Increasing the state size of linear attention improves recall performance but at the cost of higher FLOPs. In this work, we introduce Sparse Delta Memory (SDM), an architecture that scales the hidden state of gated linear RNNs to orders of magnitude higher capacity using a sparse addressing scheme. SDM extends the Gated DeltaNet architecture by replacing the dense key-value outer product with sparse reads and writes to a large explicit memory. We show that, under an isoFLOP constraint and with an identical number of parameters, a higher state memory capacity significantly improves performance on in-context learning and long-context retrieval tasks. Moreover, by learning the initial state of the SDM memory and therefore using it as a parametric memory, we show that the model further improves on a wide range of common-knowledge and reasoning tasks.

Related papers

• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion [61.57938553036056]
  ARL2 is a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state.<n>We decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame recurrent linear branch that maintains a fixed-size state for streaming context.<n>With 75% of layers replaced by hybrid linear attention, the model achieves up to 2.26 wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.
  arXiv  Detail & Related papers  (2026-05-15T19:33:45Z)
• MSA: Memory Sparse Attention for Efficient End-to-End Memory Model Scaling to 100M Tokens [38.762431789628074]
  We present Memory Sparse Attention (MSA), an end-to-end trainable, efficient, and massively scalable memory model framework.<n>MSA achieves linear complexity in both training and inference while maintaining exceptional stability.<n>We also propose Memory Interleaving to facilitate complex multi-hop reasoning across scattered memory segments.
  arXiv  Detail & Related papers  (2026-03-06T02:29:54Z)
• RAM-Net: Expressive Linear Attention with Selectively Addressable Memory [11.262593123857995]
  RAM-Net is a novel architecture designed to bridge the gap between the representational capacity of full attention and the memory efficiency of linear models.<n>The core of RAM-Net maps inputs to high-dimensional sparse vectors serving as explicit addresses, allowing the model to selectively access a massive memory state.
  arXiv  Detail & Related papers  (2026-02-12T13:55:29Z)
• Scaling Linear Attention with Sparse State Expansion [62.749291436866606]
  Transformer architecture struggles with long-context scenarios due to quadratic computation and linear memory growth.<n>We propose two key innovations to achieve more effective context compression.<n>First, we introduce a row-sparse update formulation for linear attention by conceptualizing state updating as information classification.<n>Second, we present Sparse State Expansion (SSE) within the sparse framework, which expands the contextual state into multiple partitions.
  arXiv  Detail & Related papers  (2025-07-22T13:27:31Z)
• Quantifying Memory Utilization with Effective State-Size [73.52115209375343]
  We develop a measure of textitmemory utilization'<n>This metric is tailored to the fundamental class of systems with textitinput-invariant and textitinput-varying linear operators
  arXiv  Detail & Related papers  (2025-04-28T08:12:30Z)
• CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation [63.65323577445951]
  We propose a novel approach called Cache Sparse Representation (CSR)<n>CSR transforms the dense Key-Value cache tensor into sparse indexes and weights, offering a more memory-efficient representation during LLM inference.<n>Our experiments demonstrate CSR achieves performance comparable to state-of-the-art KV cache quantization algorithms.
  arXiv  Detail & Related papers  (2024-12-16T13:01:53Z)
• B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memory [91.81390121042192]
  We develop a class of models called B'MOJO to seamlessly combine eidetic and fading memory within an composable module. B'MOJO's ability to modulate eidetic and fading memory results in better inference on longer sequences tested up to 32K tokens.
  arXiv  Detail & Related papers  (2024-07-08T18:41:01Z)
• CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory [38.429707659685974]
  Large Language Models (LLMs) struggle to handle long input sequences due to high memory and runtime costs. We introduce an associative memory module which can be coupled to any pre-trained (frozen) attention-based LLM without re-training. This architecture, which we call CAMELoT, demonstrates superior performance even with a tiny context window of 128 tokens.
  arXiv  Detail & Related papers  (2024-02-21T01:00:17Z)
• Memformer: A Memory-Augmented Transformer for Sequence Modeling [55.780849185884996]
  We present Memformer, an efficient neural network for sequence modeling. Our model achieves linear time complexity and constant memory space complexity when processing long sequences.
  arXiv  Detail & Related papers  (2020-10-14T09:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.