Maglev: Sliding Recurrent Memory

• URL: http://arxiv.org/abs/2608.02870v2
• Date: Wed, 05 Aug 2026 16:58:35 GMT
• Title: Maglev: Sliding Recurrent Memory
• Abstract summary: ours is a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention.<n>We train ours with a memory consistency loss that aligns $m_t$ with $m'_t$, allowing inference to use $P$ alone.
• Score: 8.788753986099902
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce \ours{}, a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention while remaining parallelizable during training. \ours{} consists of two coupled models: a prefiller $Q$, which leverages full attention\footnote{In practice, we use interleaved full and sliding-window attention for $Q$, as this yields stronger performance. The essential requirement is that $Q$ be more expressive than $P$, with access to the full history.} to produce memory targets $m'_t$, and a decoder $P$, which uses only sliding-window attention and recurrent K/V injection to produce decoder memories $m_t$ for next-token prediction. We train \ours{} with a memory consistency loss that aligns $m_t$ with $m'_t$, allowing inference to use $P$ alone. Empirically, \ours{} improves validation loss and downstream pretraining benchmarks over sliding-window and latent recurrent transformer baselines. Moreover, sharing parameters between $P$ and $Q$ reduces parameter memory while preserving most of the gains.

Related papers

• MoNe: Modular Neural Memory for Efficient Long Context Inference [23.94676543840457]
  MoNe is a lightweight modular neural memory that attaches to any frozen pretrained Transformer.<n>MoNe reads context in fixed-size segments via test-time learning of fast-weight neural memory networks.<n>MoNe generates keys and values from the query tokens alone, with no context tokens re-read.
  arXiv  Detail & Related papers  (2026-08-18T10:28:23Z)
• Caching for the Future: Scrub Jay Episodic Memory Principles for Agent Memory Systems [5.321970297768025]
  Existing memory architectures treat all memories as equally persistent and systematically contaminate retrieved context with outdated facts.<n>We show that per-memory, type-conditioned temporal decay, can be operationalized as an auto-classified coefficient $_i$ in an external LLM-agent memory store.<n>We introduce the Temporal Generalization Test (TGT), a benchmark with held-out retention intervals and a Generalization Gap (GenGap) metric.
  arXiv  Detail & Related papers  (2026-08-05T12:12:44Z)
• Concurrent Image Understanding and Generation: Self-Correcting Coupled Markov Jump Processes [70.61868608402723]
  We introduce $textbfSelf-Correcting Coupled Markov Jump Processes (SC-CMJP)<n>In conjunction with SC-CMJP, we introduce $textttCO_texttt2textttJump$, a novel training-free single-pass sampler for joint multimodal geneneration.<n>For training and evaluation purposes, we have created and will release three large-scale joint multimodal generation corpora.
  arXiv  Detail & Related papers  (2026-07-14T18:39:29Z)
• Towards a Dynamic and Fixed-budget Memory Bank for Efficient Streaming Video Understanding [56.17043150838547]
  CausalMem is dedicated to constructing a dynamic visual memory update mechanism.<n>It estimates the redundancy of visual tokens and updates the memory bank via an online semantic basis.<n>It achieves more than textbf20$times$ visual token compression ratio and only occupies about textbf82 MB storage.
  arXiv  Detail & Related papers  (2026-06-24T10:11:08Z)
• AdaMem: Learning What to Remember for Personalized Long-Horizon LLM Agents [79.03892269184145]
  We argue that what is worth remembering is role-dependent, and propose textbfAdaMem (Adaptive Memory)<n>AdaMem improves QA accuracy by up to textbf+9.0% over the uniform Mem0 baseline while shrinking memory volume by textbf9%.
  arXiv  Detail & Related papers  (2026-06-19T06:35:52Z)
• Enhancing Software Engineering Through Closed-Loop Memory Optimization [50.418699819003486]
  We introduce ours, a closed-loop framework for memory augmentation in software engineering (SE) agents.<n>ours grounds memory utility in textitvalidated downstream impact, establishing utility as a task-agnostic textbf benchmark and an annotation-free textbfoptimization signal.<n>Results demonstrate that ours consistently improves SE agents across settings, achieving absolute gains of up to $uparrow5.25%$ in success rate and $uparrow4.63%$ in resolve efficiency.
  arXiv  Detail & Related papers  (2026-06-04T03:17:21Z)
• Tensor Cache: Eviction-conditioned Associative Memory for Transformers [20.67103891489219]
  Cache pairs sliding-window softmax attention as a first-level cache (L1) with a fixed-size outer-product fast-weight memory as a second-level cache fed by KV pairs evicted from the window.<n>Recent tokens remain in exact local attention; evicted pairs are compressed into a per-layer matrix $A$ and read by future queries through a single matrix multiplication.
  arXiv  Detail & Related papers  (2026-05-21T00:21:51Z)
• Inhibitory Cross-Talk Enables Functional Lateralization in Attention-Coupled Latent Memory [0.0]
  We present a memory-augmented transformer in which attention serves simultaneously as a retrieval, consolidation, and write-back operator.<n>We partition the memory into lateralized left and right banks coupled through a sign-controlled cross-talk matrix $W_s$, and show that the sign of this coupling is decisive for specialization.
  arXiv  Detail & Related papers  (2026-02-27T08:58:51Z)
• Remember Me, Refine Me: A Dynamic Procedural Memory Framework for Experience-Driven Agent Evolution [52.76038908826961]
  We propose $textbfReMe$ ($textitRemember Me, Refine Me$) to bridge the gap between static storage and dynamic reasoning.<n>ReMe innovates across the memory lifecycle via three mechanisms: $textitmulti-faceted distillation$, which extracts fine-grained experiences.<n>Experiments on BFCL-V3 and AppWorld demonstrate that ReMe establishes a new state-of-the-art in agent memory system.
  arXiv  Detail & Related papers  (2025-12-11T14:40:01Z)
• QKV Projections Require a Fraction of Their Memory [11.048086031066623]
  We propose a novel tensor compression technique that reduces memory consumption of the $Q,K,V$ projections in attention layers by a factor of up to $times 512$.<n>PAMM is fully composable with efficient attention techniques such as FlashAttention, making it a practical and complementary method for memory-efficient LLM training.
  arXiv  Detail & Related papers  (2025-06-03T14:37:17Z)
• Scaling Embedding Layers in Language Models [61.939921364422936]
  $SCONE$ is a new method for extending input embedding layers to enhance language model performance.<n>$SCONE$ retains the original vocabulary while introducing embeddings for a set of frequent n-grams.<n>These embeddings provide contextualized representation for each input token and are learned with a separate model during training.<n>$SCONE$ enables two new scaling strategies: increasing the number of n-gram embeddings and scaling the model used to learn them, both while maintaining fixed accelerator usage during inference.
  arXiv  Detail & Related papers  (2025-02-03T18:59:32Z)
• Sub-Linear Memory: How to Make Performers SLiM [38.068090269482425]
  vanilla Transformers require $O(L2)$ in serial time and memory as functions of input length $L$. Recent works proposed various linear self-attention mechanisms, scaling only as $O(L)$ for serial computation. We observe a remarkable computational flexibility: forward and backward propagation can be performed with no approximations using sublinear memory.
  arXiv  Detail & Related papers  (2020-12-21T13:56:04Z)
• GMAT: Global Memory Augmentation for Transformers [45.584411593847406]
  We propose to augment sparse Transformer blocks with a dense attention-based $textitglobal memory$ of length $M$ ($ll L$) Our augmentation has a manageable $O(Mcdot(L+M))$ memory overhead, and can be seamlessly integrated with prior sparse solutions.
  arXiv  Detail & Related papers  (2020-06-05T07:50:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.