MARCH: Scaling Recurrent Memory with Content-Routed State Anchors

• URL: http://arxiv.org/abs/2608.12435v1
• Date: Wed, 12 Aug 2026 13:45:01 GMT
• Title: MARCH: Scaling Recurrent Memory with Content-Routed State Anchors
• Abstract summary: Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length.<n>Recurrent alternatives offer efficient decoding by compressing the entire history into a fixed-size state.<n>We introduce Memory-Anchor Routing across Context History (MARCH), a network architecture that effectively scales state-space models beyond a fixed-size dimension.
• Score: 41.39470291507015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Transformers owe much of their strong long-context retrieval capability to a token-level memory that grows with context length. This flexibility, however, incurs a quadratic computation complexity during training and a key--value cache that grows linearly during autoregressive inference. Recurrent alternatives offer efficient decoding by compressing the entire history into a fixed-size state, but often underperform on recall-intensive tasks since earlier associations usually get overwritten by subsequent updates, and only the most recent contextual information is retained. In this paper, we introduce Memory-Anchor Routing across Context History (MARCH), a network architecture that effectively scales state-space models beyond a fixed-size dimension, while maintaining computational efficiency over long-sequences. MARCH periodically caches cumulative recurrent-state checkpoints as state anchors and associates each anchor with a compact, content-conditioned anchor key. This lets MARCH maintain a memory bank, which can grow as context length increases, providing a controllable trade-off between historical resolution and memory cost. At each token, MARCH produces an anchor query to attend all causally available state anchors, and the output is calculated as an attention-style aggregation over all historical anchors along the current state. We show that after standard pretraining, MARCH consistently outperforms multiple linear attention variants across commonsense reasoning, LongBench, and in-context retrieval. These results demonstrate that content-routed state caching substantially strengthens recurrent long-range memory while preserving its native computation path.

Related papers

• LayerRecall: A State-Conditioned Memory Router for Long-Horizon Consistency in Video Generation [22.878882105342182]
  Autoregressive video diffusion produces chunks from a bounded recent context.<n>Recency-based caching evicts historical cues needed when subjects, objects, scenes, or attributes reappear.<n>LayerRecall retrieves relevant historical K/V states and injects them only into backbone-specific memory-sensitive layers.
  arXiv  Detail & Related papers  (2026-08-28T15:49:13Z)
• Raven: High-Recall Sequence Modeling with Sparse Memory Routing [88.47618436676748]
  We introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing.<n>Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade.
  arXiv  Detail & Related papers  (2026-07-28T07:04:32Z)
• FadeMem: Distance-Aware Memory Consolidation for Autoregressive Video Diffusion [59.207505503284715]
  FadeMem organizes historical KV blocks into a temporal hierarchy under a fixed cache budget.<n>New history is inserted as fine-grained entries, while older adjacent entries are progressively merged.<n>Experiments show improved subject consistency, background stability, and temporal coherence over existing bounded-cache strategies.
  arXiv  Detail & Related papers  (2026-06-09T10:22:18Z)
• MemForest: An Efficient Agent Memory System with Hierarchical Temporal Indexing [33.94614583313314]
  We present MemForest, a memory framework that reformulates agent memory as a write-efficient temporal data management problem.<n>MemForest breaks the sequential bottleneck via parallel chunk extraction, decoupling memory construction into concurrent, independent operations.<n>We evaluate MemForest on two long-context memory benchmarks, LongMemEval-S and LoCoMo.
  arXiv  Detail & Related papers  (2026-05-16T13:11:47Z)
• StructKV: Preserving the Structural Skeleton for Scalable Long-Context Inference [38.75965915663233]
  We propose StructKV, a structure-aware KV cache compression framework.<n>First, Global In-Degree Centrality aggregates attention patterns across the network depth to identify global information hubs.<n>Second, Dynamic Pivot Detection utilizes information-theoretic metrics to adaptively locate the optimal layer for compression.
  arXiv  Detail & Related papers  (2026-04-08T07:10:35Z)
• Learning to Remember: End-to-End Training of Memory Agents for Long-Context Reasoning [18.621823772319154]
  We propose an end-to-end reinforcement learning framework that unifies memory operations and question answering within a single policy.<n>UMA maintains a dual memory representation: a compact core summary for global context and a structured Memory Bank that supports explicit CRUD.<n>Across 13 datasets spanning Ledger-QA, Test-Time Learning, and Accurate Retrieval, UMA substantially outperforms long-context and RAG baselines on dynamic reasoning and learning tasks.
  arXiv  Detail & Related papers  (2026-02-13T16:54:23Z)
• AMA: Adaptive Memory via Multi-Agent Collaboration [54.490349689939166]
  We propose Adaptive Memory via Multi-Agent Collaboration (AMA), a novel framework that leverages coordinated agents to manage memory across multiple granularities.<n>AMA significantly outperforms state-of-the-art baselines while reducing token consumption by approximately 80% compared to full-context methods.
  arXiv  Detail & Related papers  (2026-01-28T08:09:49Z)
• MemFlow: Flowing Adaptive Memory for Consistent and Efficient Long Video Narratives [54.07515675393396]
  Existing solutions maintain the memory by compressing historical frames with predefined strategies.<n>We propose MemFlow to address this problem.<n>MemFlow achieves outstanding long-context consistency with negligible burden.
  arXiv  Detail & Related papers  (2025-12-16T18:59:59Z)
• Mixture of Contexts for Long Video Generation [72.96361488755986]
  We recast long-context video generation as an internal information retrieval task.<n>We propose a simple, learnable sparse attention routing module, Mixture of Contexts (MoC), as an effective long-term memory retrieval engine.<n>As we scale the data and gradually sparsify the routing, the model allocates compute to salient history, preserving identities, actions, and scenes over minutes of content.
  arXiv  Detail & Related papers  (2025-08-28T17:57:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.