The Query Knows What to Forget: A Second Erase Direction for Linear Attention

• URL: http://arxiv.org/abs/2608.13668v1
• Date: Thu, 13 Aug 2026 18:04:53 GMT
• Title: The Query Knows What to Forget: A Second Erase Direction for Linear Attention
• Abstract summary: Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase vector from the key of the current token.<n>We introduce the Query-derived Erase Direction (QED)<n>It improves retrieval at every length past the training window, and it about doubles the usable context length on S-NIAH-1.
• Score: 4.500637354443275
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Linear attention keeps a state of fixed size. At long context, many stored items share this state, and interference between them degrades retrieval. Gated DeltaNet-2 (GDN-2), like every delta-rule model before it, derives its erase vector from the key of the current token. However, the interference in its reads is measured through the query, and the erase step cannot reach it. We introduce the Query-derived Erase Direction (QED). QED adds a second erase direction derived from the query and orthogonal to the key. In the fast-weight view, a key-directed delta edit cannot change the key-orthogonal part of a read. It uses the editable part to cancel old-state content measured along the query. It also improves retrieval at every length past the training window, and it about doubles the usable context length on S-NIAH-1.

Related papers

• Erase-then-Delta Attention: Decoupling Erase and Write Addresses in Delta-Rule Linear Attention [61.96880087320455]
  Delta-rule linear attention improves recurrent memory updates by correcting what is already stored at the current write address.<n>We propose Erase-then-Delta Attention (EDA), a memory update rule that decouples where to erase from where to write.
  arXiv  Detail & Related papers  (2026-06-25T03:12:19Z)
• Don't Read Everything: A Curvature-Conditioned Query for Linear Attention [42.02790959450039]
  Linear attention reduces the quadratic cost of softmax attention by maintaining a recurrent fast-weight state.<n>Existing remedies act on the write side of memory through gating, delta updates, or kernel feature maps.<n>We borrow one specific piece of softmax's geometry to construct a cheap read-time contraction of the query.<n>We call this mechanism Curvature-Conditioned Query (CCQ)<n>It improves perplexity, zero-shot downstream accuracy, S-NIAH retrieval at and beyond the training context, length-extrapolation perplexity from 4K to 20K, and Long
  arXiv  Detail & Related papers  (2026-05-31T15:25:42Z)
• Gated DeltaNet-2: Decoupling Erase and Write in Linear Attention [81.79922329750674]
  Linear attention replaces the cache of softmax attention with a fixed-size recurrent state, reducing sequence mixing to linear time and decoding to constant memory.<n>We introduce Gated DeltaNet-2, which generalizes both Gated DeltaNet and Kimi Delta Attention.<n>Gated DeltaNet-2 achieves the strongest overall results among Mamba Gated DeltaNet, KDA, and Mamba-3 variants across language modeling, commonsense reasoning, and retrieval.
  arXiv  Detail & Related papers  (2026-05-21T17:44:57Z)
• HoReN: Normalized Hopfield Retrieval for Large-Scale Sequential Model Editing [12.303016850029683]
  Large language models encode vast factual knowledge that inevitably becomes outdated or incorrect after deployment.<n>One line of work installs new facts by directly modifying base weights through locate-then-edit procedures.<n>A complementary line leaves base weights intact and routes edits through external memory, but it faces routing challenges.<n>We propose HoReN, a codebook-based editor with enhanced routing built on three ideas.
  arXiv  Detail & Related papers  (2026-05-02T15:51:31Z)
• Residual Stream Duality in Modern Transformer Architectures [9.910562011343009]
  Recent work has made clear that the residual pathway is not mere optimization plumbing; it is part of the model's representational machinery.<n>We argue that the cleanest way to organize this design space is through a two-axis view of the Transformer.
  arXiv  Detail & Related papers  (2026-03-17T00:56:29Z)
• TRA: Better Length Generalisation with Threshold Relative Attention [58.64717643300818]
  Transformers struggle with length generalisation, displaying poor performance even on basic tasks.<n>We test whether these limitations can be explained through two key failures of the self-attention mechanism.<n>We show how the attention mechanism with these two mitigations in place can substantially improve the generalisation capabilities of decoder only transformers.
  arXiv  Detail & Related papers  (2025-03-29T18:06:28Z)
• Attention Map Guided Transformer Pruning for Edge Device [98.42178656762114]
  Vision transformer (ViT) has achieved promising success in both holistic and occluded person re-identification (Re-ID) tasks. We propose a novel attention map guided (AMG) transformer pruning method, which removes both redundant tokens and heads. Comprehensive experiments on Occluded DukeMTMC and Market-1501 demonstrate the effectiveness of our proposals.
  arXiv  Detail & Related papers  (2023-04-04T01:51:53Z)
• Query Resolution for Conversational Search with Limited Supervision [63.131221660019776]
  We propose QuReTeC (Query Resolution by Term Classification), a neural query resolution model based on bidirectional transformers. We show that QuReTeC outperforms state-of-the-art models, and furthermore, that our distant supervision method can be used to substantially reduce the amount of human-curated data required to train QuReTeC.
  arXiv  Detail & Related papers  (2020-05-24T11:37:22Z)
• On the Arbitrary-Oriented Object Detection: Classification based Approaches Revisited [94.5455251250471]
  We first show that the boundary problem suffered in existing dominant regression-based rotation detectors, is caused by angular periodicity or corner ordering. We transform the angular prediction task from a regression problem to a classification one. For the resulting circularly distributed angle classification problem, we first devise a Circular Smooth Label technique to handle the periodicity of angle and increase the error tolerance to adjacent angles.
  arXiv  Detail & Related papers  (2020-03-12T03:23:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.