Raven: High-Recall Sequence Modeling with Sparse Memory Routing

• URL: http://arxiv.org/abs/2607.25357v1
• Date: Tue, 28 Jul 2026 07:04:32 GMT
• Title: Raven: High-Recall Sequence Modeling with Sparse Memory Routing
• Abstract summary: We introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing.<n>Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade.
• Score: 88.47618436676748
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-context recall in linear-time sequence models highlights a tradeoff in how they write to memory. State-based linear models, such as state-space models (SSMs) and linear Transformers, write densely, updating the entire state for each newly arrived token, which leads to interference and makes specific past tokens hard to recover. Sliding-window attention (SWA) exhibits the opposite behavior: it writes sparsely by storing explicit token representations, but only within a fixed window, so recall drops once the relevant token is evicted. Interpolating between these models, we introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing. This lets Raven mitigate SWA's position-based overwriting and hard eviction while reducing interference from dense state updates in SSMs, thereby preserving long-range content much more effectively. Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade. It remains effective when extrapolating to context lengths as large as 16x its training length, with similar gains in hybrid architectures.

Related papers

• MARCH: Scaling Recurrent Memory with Content-Routed State Anchors [41.39470291507015]
  Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length.<n>Recurrent alternatives offer efficient decoding by compressing the entire history into a fixed-size state.<n>We introduce Memory-Anchor Routing across Context History (MARCH), a network architecture that effectively scales state-space models beyond a fixed-size dimension.
  arXiv  Detail & Related papers  (2026-08-12T13:45:01Z)
• Remembering Distinct Items, Not Tokens: A Learnable Dirichlet-Process Cache Between State-Space Models and Attention [0.48342038441006796]
  We study a sparse cache that allocates a slot only when an input is novel, so its size tracks the number of distinct items rather than the number of tokens.<n>We develop it in two forms, a static cache with a fixed concentration and a surprise-adaptive variant whose concentration follows the recent novelty rate.
  arXiv  Detail & Related papers  (2026-07-10T18:28:21Z)
• SHARP: Sleep-based Hierarchical Accelerated Replay for Long Range Non-Stationary Temporal Pattern Recognition [7.555597360852762]
  SHARP (Sleep-based Hierarchical Accelerated Replay) is a framework that decomposes temporal learning into two complementary components.<n>Inspired by the accelerated replay observed in rodents during slow-wave sleep, SHARP incorporates offline (sleep) phases in which temporally structured memory traces are replayed in an accelerated form.
  arXiv  Detail & Related papers  (2026-05-30T13:55:02Z)
• Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference [58.46323651532913]
  We study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights.<n>During sleep, the model performs $N$ offline recurrent passes over the accumulated context and updates the fast weights in its state-space model.<n>We show that increasing sleep duration $N$ for our models improves performance, with the largest gains on examples that require deeper reasoning.
  arXiv  Detail & Related papers  (2026-05-25T17:55:39Z)
• SAM: State-Adaptive Memory for Long-Horizon Reasoning Agent [51.274445160155864]
  Long-horizon agentic reasoning requires large language models to act over long interaction histories containing thoughts, tool calls, observations, and partial conclusions.<n>Existing approaches address this difficulty by truncating the interaction history, compressing it into shorter surrogates, or retrieving selected parts of it for reuse.<n>We propose State-Adaptive Memory, a standalone framework that consolidates ongoing interaction into compact memory cues while preserving raw trajectory pages for intent-driven recall.
  arXiv  Detail & Related papers  (2026-05-23T08:37:16Z)
• Attend Locally, Remember Linearly: Linear Attention as Cross-Frame Memory for Autoregressive Video Diffusion [61.57938553036056]
  ARL2 is a hybrid attention module that replaces quadratic cross-frame attention with a fixed-size recurrent state.<n>We decompose self-attention into two branches: an intra-frame softmax branch for spatial detail and local dependencies, and an inter-frame recurrent linear branch that maintains a fixed-size state for streaming context.<n>With 75% of layers replaced by hybrid linear attention, the model achieves up to 2.26 wall-clock speedup and 54% memory reduction, while maintaining comparable quality with improving temporal consistency.
  arXiv  Detail & Related papers  (2026-05-15T19:33:45Z)
• Adaptive Memory Decay for Log-Linear Attention [1.0099625992507715]
  Sequence models face a fundamental tradeoff between memory capacity and computational efficiency.<n>Log-linear attention navigates this tradeoff by organizing memory across a Fenwick tree hierarchy.<n>We propose learning directly from the input via a lightweight two-layer, producing per-token, per-level decay that adapts to content rather than position.
  arXiv  Detail & Related papers  (2026-05-07T21:05:28Z)
• Efficient Sparse Selective-Update RNNs for Long-Range Sequence Modeling [16.27516369166629]
  Selective-Update RNNs (suRNNs) are a non-linear architecture that learns to preserve its memory when the input is redundant.<n>By using a neuron-level binary switch that only opens for informative events, suRNNs decouple the recurrent updates from the raw sequence length.<n>Experiments on the Long Range Arena, WikiText, and other synthetic benchmarks show that suRNNs match or exceed the accuracy of much more complex models.
  arXiv  Detail & Related papers  (2026-02-11T14:50:07Z)
• Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regression [53.48692193399171]
  Gated KalmaNet (GKA) is a layer that reduces the gap by accounting for the full past when predicting the next token.<n>We solve an online ridge regression problem at test time, with constant memory and linear compute cost in the sequence length.<n>On long-context, GKA excels at real-world RAG and LongQA tasks up to 128k tokens, achieving more than $10$% relative improvement over other fading memory baselines.
  arXiv  Detail & Related papers  (2025-11-26T03:26:37Z)
• CItruS: Chunked Instruction-aware State Eviction for Long Sequence Modeling [52.404072802235234]
  We introduce Chunked Instruction-aware State Eviction (CItruS), a novel modeling technique that integrates the attention preferences useful for a downstream task into the eviction process of hidden states. Our training-free method exhibits superior performance on long sequence comprehension and retrieval tasks over several strong baselines under the same memory budget.
  arXiv  Detail & Related papers  (2024-06-17T18:34:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.