Memory-Managed Long-Context Attention: A Preliminary Study of Editable Request-Local Memory

• URL: http://arxiv.org/abs/2606.28876v1
• Date: Sat, 27 Jun 2026 11:38:43 GMT
• Title: Memory-Managed Long-Context Attention: A Preliminary Study of Editable Request-Local Memory
• Abstract summary: We study memory-managed long-context attention, a research route that separates a fast recurrent or sparse backbone from explicit editable request-local memory slots.<n>Across structured synthetic tasks, token/chunk/sequence bridges, generated natural language, and local frozen-model diagnostics, pure fixed-state or pure sparse methods fail some overwrite, version, anti-pollution, or no-write-signal cases.
• Score: 0.5371337604556311
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Long-context language models often conflate two different goals: compressing history into an efficient state, and maintaining reliable long-term memory. Linear, recurrent, and sparse attention reduce the cost of processing long sequences, but they do not by themselves specify when a fact should be written, overwritten, protected from distractors, or discarded. We study memory-managed long-context attention, a research route that separates a fast recurrent or sparse backbone from explicit editable request-local memory slots and query-time sparse fallback. Across structured synthetic tasks, token/chunk/sequence bridges, generated natural language, and local frozen-model diagnostics, pure fixed-state or pure sparse methods fail some overwrite, version, anti-pollution, or no-write-signal cases, while a hybrid covers both routes. A small 2,097,152-token mechanism stress test reaches 50/50 pooled accuracy with 2-132 active chunks. A 2.74M-parameter minimal causal event-token model reaches 595/600 with lite write supervision, supporting proof of trainability rather than scale. A six-family frozen-hidden-state bridge reaches 1079/1080 controlled pointer accuracy, but it uses generator-provided integer key IDs and separately encoded canonical key strings; it is an oracle-metadata probe, not open-text entity resolution. Local non-leaderboard RULER 4K diagnostics remain close to full context, whereas a 33-record LongBench v1 16K subset shows that naive lexical selection is not general. The evidence separates three claims: controlled slot lifecycle is feasible, sparse fallback is needed when writes lack future-query signals, and learned open-domain selection remains the main architectural bottleneck. We do not claim a final generative architecture, global slot-trajectory convergence, or systems superiority.

Related papers

• EM^2Mem: Event-Centric Multimodal Memory for Large Language Models [52.3165591032625]
  We propose EM2Mem, an event-centric multimodal memory framework that aligns heterogeneous evidence to event anchors during memory construction.<n>Across three long-video QA benchmarks, EM2Mem improves average accuracy over the strongest memory baseline by 2.0, 2.4, and 3.7 points.
  arXiv  Detail & Related papers  (2026-09-01T01:38:41Z)
• Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings [0.0]
  A seeded life-script sampler emits facts with validity intervals volatility classes, and source channels before any text exists.<n>A verifier confirms every planted fact and questions are instantiated mechanically from the script.<n>The synthetic, fictionalized corpus embeds features absent from the benchmarks we survey.
  arXiv  Detail & Related papers  (2026-07-24T04:19:45Z)
• MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations [50.24315431387575]
  We introduce MemOps, a benchmark that reformulates conversational memory as a sequence of lifecycle operations.<n>MemOps represents each memory event with a structured trace specifying its trigger, target, scope, state transition, and supporting evidence.<n>Across long-context, retrieval-based, parametric and managed-memory systems, MemOps disentangles failure modes that final-answer accuracy alone conceals.
  arXiv  Detail & Related papers  (2026-07-14T15:33:44Z)
• ConvMemory v3: A Validity Context Layer for Conversational Memory via Target-Conditioned Relation Verification [0.0]
  ConvMemory v3 adds a validity context layer that detects and surfaces this update evidence.<n>The core mechanism is a dual-evidence gate that conditions a relation judgment on the specific target proposition.<n>On a synthetic multi-hop validity benchmark the gate reaches 90.12% +/- 1.73 accuracy.
  arXiv  Detail & Related papers  (2026-06-25T08:36:43Z)
• Efficient and Trainable Language Model Test-Time Scaling via Local Branch Routing [65.59464371673623]
  Local Branch Routing (LBR) is a token-level test-time scaling framework.<n>It expands a small local lookahead tree, forwards all sampled branches through the language model, and uses a lightweight router to select the depth-1 subtree to commit.<n>LBR allows each token decision to use evidence beyond the root next-token distribution while avoiding full solution-level search.
  arXiv  Detail & Related papers  (2026-06-24T03:42:44Z)
• Mitigating Provenance-Role Collapse in Long-Term Agents via Typed Memory Representation [33.47493976010677]
  We propose MemIR, a typed Memory Intermediate Representation that operationalizes source monitoring as a structural constraint.<n>MemIR writes long-term memory into grounded atoms that separate raw evidence, retrieval cues, and truth-bearing claims.<n>Experiments on LoCoMo and BEAM-100K demonstrate that MemIR consistently outperforms existing memory baselines.
  arXiv  Detail & Related papers  (2026-05-25T13:56:31Z)
• MemX: A Local-First Long-Term Memory System for AI Assistants [0.0]
  We present MemX, a local-first long-term memory system for AI with stability-oriented retrieval design.<n>MemX is implemented in Rust on top of lib and an OpenAI-compatible embedding API, providing persistent, searchable, and explainable memory for conversational agents.
  arXiv  Detail & Related papers  (2026-03-17T06:39:54Z)
• S$^3$-Attention:Attention-Aligned Endogenous Retrieval for Memory-Bounded Long-Context Inference [11.779449360037518]
  We present S3-Attention, a memory-first inference-time framework that treats long-context processing as attention-aligned endogenous retrieval.<n>S3-Attention decodes transient key and query projections into top-k sparse feature identifiers using lightweight sparse autoencoders.<n>It constructs a CPU-based inverted index mapping features to token positions or spans during a single streaming scan.
  arXiv  Detail & Related papers  (2026-01-25T05:25:22Z)
• Short-Context Dominance: How Much Local Context Natural Language Actually Needs? [48.429870236229696]
  We measure the minimum context length needed to reproduce accurate full-context predictions.<n>For sequences with 1-7k tokens from long-context documents, we consistently find that 75-80% require only the last 96 tokens at most.<n>We introduce a practical proxy to MCL, called Distributionally Aware MCL (DaMCL), that does not require knowledge of the actual next-token.
  arXiv  Detail & Related papers  (2025-12-08T22:25:00Z)
• On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attention [53.22963042513293]
  Large language models (LLMs) excel at capturing global token dependencies via self-attention but face prohibitive compute and memory costs on lengthy inputs.<n>We first propose dual-state linear attention (A), a novel design that maintains two hidden states-one for preserving historical context and one for tracking recencythereby mitigating the short-range bias typical of linear-attention architectures.<n>We introduce DSLA-Serve, an online adaptive distillation framework that progressively replaces Transformer layers DSLA layers at inference time, guided by a sensitivity-based layer ordering.
  arXiv  Detail & Related papers  (2025-06-11T01:25:06Z)
• LongProc: Benchmarking Long-Context Language Models on Long Procedural Generation [62.03873304893779]
  LongProc (Long Procedural Generation) is a new benchmark for evaluating long-context language models.<n>It requires the integration of highly dispersed information and long-form generation.<n>We evaluate 23 LCLMs on LongProc at three difficulty levels, with the maxi- mum number of output tokens set at 500, 2K, and 8K.
  arXiv  Detail & Related papers  (2025-01-09T18:16:55Z)
• Training-Free Exponential Context Extension via Cascading KV Cache [49.608367376911694]
  We introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens.<n>Our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens.
  arXiv  Detail & Related papers  (2024-06-24T03:59:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.