Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention

• URL: http://arxiv.org/abs/2606.26560v1
• Date: Thu, 25 Jun 2026 03:12:19 GMT
• Title: Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention
• Abstract summary: Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address.<n>We propose Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write.
• Score: 61.96880087320455
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address before writing new content. However, the active correction is still anchored to that same write address. As a result, stale information stored at a different address cannot be actively removed before new content is written elsewhere. We propose Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write. The key insight is that recurrent memory models should not only correct the current write, but also selectively suppress outdated memory at an independently chosen address. Concretely, our method first applies a targeted erase step along a learned erase direction, and then performs the standard delta-style corrective write along the current write direction. This preserves the corrective behavior of delta-rule updates while expanding their memory-management capacity. Language-model pretraining experiments across dense 2.5B and MoE 25B-A2.8B model families show that EDA performs best in both settings. The gain persists after 80B-token long-context midtraining of the MoE models, where EDA also performs best in long-context evaluations from 4k to 128k contexts. A compact update analysis and memory-state probes suggest why: EDA keeps the delta-rule corrective write intact while allocating an additional cleanup path most strongly when passive decay is weak. These results suggest that recurrent memory models should decide not only what to write, but also what stale information to erase and where.

Related papers

• The Query Knows What to Forget: A Second Erase Direction for Linear Attention [4.500637354443275]
  Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase vector from the key of the current token.<n>We introduce the Query-derived Erase Direction (QED)<n>It improves retrieval at every length past the training window, and it about doubles the usable context length on S-NIAH-1.
  arXiv  Detail & Related papers  (2026-08-13T18:04:53Z)
• TARL: Transaction-Aware Reliable Ledgers for Executable Memory Management in Long-Term Agents [20.123133746969398]
  We introduce TARL, a memory state update framework that maps each statement to one of five executable actions.<n> TARL identifies the affected memory, resolves its temporal scope, compares source reliability, and updates accepted, pending, and rejected ledgers.<n>We also introduce TARL-Mem, a benchmark with fine-grained action labels and next-state targets.
  arXiv  Detail & Related papers  (2026-08-04T14:02:55Z)
• Raven: High-Recall Sequence Modeling with Sparse Memory Routing [88.47618436676748]
  We introduce Raven, a linear-time sequence model that maintains a fixed set of memory slots and, at each step, decays and updates only a selected subset via learned, input-dependent routing.<n>Across recall-intensive benchmarks, Raven is competitive with or outperforms prior linear-time baselines, achieving strong long-context recall where both SWA and SSMs sharply degrade.
  arXiv  Detail & Related papers  (2026-07-28T07:04:32Z)
• The Past Is Prologue: A Plug-in Controller for Selective Updates in Sequentially Evolving LLM Memory [53.50071717647285]
  We propose Janus, a plug-in memory controller that decides whether to accept a candidate memory update or retain the previous memory.<n>Janus uses a Memory Momentum Trigger to identify suspicious deviations in the memory-update trajectory.<n>Janus improves average accuracy by +2.7 to +4.6 points over the corresponding base updaters.
  arXiv  Detail & Related papers  (2026-06-30T04:33:18Z)
• AdaMem: Learning What to Remember for Personalized Long-Horizon LLM Agents [79.03892269184145]
  We argue that what is worth remembering is role-dependent, and propose textbfAdaMem (Adaptive Memory)<n>AdaMem improves QA accuracy by up to textbf+9.0% over the uniform Mem0 baseline while shrinking memory volume by textbf9%.
  arXiv  Detail & Related papers  (2026-06-19T06:35:52Z)
• Leyline: KV Cache Directives for Agentic Inference [0.11099872871193028]
  Leyline is a serving-side primitive for agentic editing.<n>A declarative directive 4-tuple separates what to edit from how to preserve position correctness.<n>A ten-line truncation rule routed through the same interface lifts agentic solve rate by +14.3 pp on debug-gym.
  arXiv  Detail & Related papers  (2026-05-31T07:13:15Z)
• Is Agent Memory a Database? Rethinking Data Foundations for Long-Term AI Agent Memory [3.263386002412657]
  Long-running AI agents need persistent memory.<n>Current agent memory systems and database paradigms treat memory as storage.<n>In our vision, long-term agent memory is a new data-management workload.
  arXiv  Detail & Related papers  (2026-05-25T18:22:42Z)
• Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention [81.79922329750674]
  Linear attention replaces the cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.<n>We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and Kimi Delta Attention.<n>Gated DeltaNet-2 achieves the strongest overall results among Mamba Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval.
  arXiv  Detail & Related papers  (2026-05-21T17:44:57Z)
• GradMem: Learning to Write Context into Memory with Test-Time Gradient Descent [10.16391634040356]
  We introduce GradMem, which writes context into memory via per-sample test-time optimization.<n> GradMem explicitly optimize a model-level self-supervised context reconstruction loss.<n>It attains competitive results on natural language tasks including bAbI and SQuAD variants.
  arXiv  Detail & Related papers  (2026-03-14T10:17:33Z)
• UPCORE: Utility-Preserving Coreset Selection for Balanced Unlearning [57.081646768835704]
  User specifications or legal frameworks often require information to be removed from pretrained models, including large language models (LLMs)<n>This requires deleting or "forgetting" a set of data points from an already-trained model, which typically degrades its performance on other data points.<n>We propose UPCORE, a method-agnostic data selection framework for mitigating collateral damage during unlearning.
  arXiv  Detail & Related papers  (2025-02-20T22:51:10Z)
• Language Modeling with Editable External Knowledge [90.7714362827356]
  This paper introduces ERASE, which improves model behavior when new documents are acquired. It incrementally deletes or rewriting other entries in the knowledge base each time a document is added. It improves accuracy relative to conventional retrieval-augmented generation by 7-13% (Mixtral-8x7B) and 6-10% (Llama-3-8B) absolute.
  arXiv  Detail & Related papers  (2024-06-17T17:59:35Z)
• LaMemo: Language Modeling with Look-Ahead Memory [50.6248714811912]
  We propose Look-Ahead Memory (LaMemo) that enhances the recurrence memory by incrementally attending to the right-side tokens. LaMemo embraces bi-directional attention and segment recurrence with an additional overhead only linearly proportional to the memory length. Experiments on widely used language modeling benchmarks demonstrate its superiority over the baselines equipped with different types of memory.
  arXiv  Detail & Related papers  (2022-04-15T06:11:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.