Forgetful Attention: A Trainable Support-Vector Memory with Certified Selection and Exact Unlearning

• URL: http://arxiv.org/abs/2607.12204v1
• Date: Mon, 13 Jul 2026 23:13:57 GMT
• Title: Forgetful Attention: A Trainable Support-Vector Memory with Certified Selection and Exact Unlearning
• Abstract summary: We introduce Support Vector Attention (SV-Attention), a memory whose weights are support coefficients of a one-class Sfp with fixed box parameter C.<n>Its active-set partition gives reserve tokens exactly zero weight, output weights; a incremental incremental solver token to recover the state produced by retraining without it under the same C.<n>We also demonstrate surgical eviction, exact editing, patient-record deletion, and a forgettable retrieval memory over real sentence embeddings.
• Score: 0.16921396880325776
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Attention can be viewed as an online learner over context, yet existing test-time memories cannot certify that dropping a token leaves outputs unchanged or delete its influence outright. We introduce Support Vector Attention (SV-Attention), a max-margin memory whose weights are support coefficients of a one-class SVM with fixed box parameter C. Its active-set partition gives reserve tokens exactly zero weight, certifying output-preserving eviction; a reversible incremental solver deletes a token to recover the state produced by retraining without it under the same C. In fp64 experiments, decrement and refit recover identical partitions whenever the optimum is unique, and their decision functions match to a median deviation of about 10^-9 (10^-13 on learned keys); the 10^-2 worst case is confined to ill-conditioned duplicates and remains below coefficient decay in every regime. The exact path reuses the maintained KKT inverse in a custom backward. Training uses a separate stabilized batched approximation and does not carry the exact-deletion certificate; it reaches 9,125 tokens/s on a 3.22M-parameter model, while remaining 35.8 times slower than an MPS softmax reference. At matched budgets, certified selection reaches 0.86 vs. 0.32 rare-item recall and retains 0.80 vs. 0.05 deterioration hours on real MIMIC-IV streams. We also demonstrate surgical forgetting, exact editing, patient-record deletion, and a forgettable retrieval memory over real sentence embeddings. On enwik8, the hybrid obtains 2.178 BPC vs. 2.383 for a matched-state sliding-window Transformer across seven seeds (8.6% paired improvement, p=0.001); a three-seed TinyStories result is directionally positive but not significant (p=0.057).

Related papers

• Adapting Without Gradients: Affine Statistics Transport and What Its Certificate Can Tell You [0.8594140167290097]
  Test-time adaptation assumes that model parameters can be updated at inference time.<n>We study adaptation when the learned model must remain frozen.<n>We introduce CASTER, a method that stores source class statistics in a discriminative subspace.
  arXiv  Detail & Related papers  (2026-08-31T21:07:36Z)
• On the Design of Qwen3.8-Next Architecture: Evaluation, Efficiency, and Training Stability [57.03393882933515]
  We describe the architecture and ablations of Qwen3.8-Flash-Next, a sparse mixture-of-experts model with 125B parameters, 6B activated per token, and 51B parameters of n-gram embedding tables held off the accelerator.<n>On fourteen pre-training benchmarks the model leads the 397B-A17B predecessor on eight and trails it on the rest by at most 2.6 points, at 1/3 the activated parameters, 1/3 the training tokens, and roughly 1/9 the training FLOPs.
  arXiv  Detail & Related papers  (2026-08-31T06:35:07Z)
• The Recall Trap: A Recall-Maximizing Retriever Configuration Reduces Issue Resolution in Fixed-Budget Code Context [45.88028371034407]
  We report a controlled case study in code repair, not a new phenomenon.<n>We toggle one flag (one-chunk-perfile deduplication) on an otherwise identical stack.<n>We map where it holds: it reverses on a lexical BM25 (3.2pp, significant cross-paradigm interaction)<n>We show that at a tight fixed budget, do not hard-deduplicate by file, and A/B packing policies against the task, not the metric flag was tuned.
  arXiv  Detail & Related papers  (2026-08-14T19:22:50Z)
• Support Operation Factorization: Compositional Readout of Frozen Vision Encoders under Controlled Interventions [87.95757610455173]
  We introduce an injectively aligned leave-one-cell-out protocol over support x operation grids and SO-OPF.<n>We separate two questions that aggregate scores conflate: whether the carrier composes held-out bindings when the grid is known, and whether that grid can be recovered from flat cell labels.
  arXiv  Detail & Related papers  (2026-08-06T15:38:55Z)
• Memoir: Should a Model Write to Its Memory While It Thinks? [0.0]
  Memoir combines per-sample fast memory, shared slow parameters, variable-depth latent recurrence, and a future-latent energy objective.<n>We compare a coupled arm against an otherwise identical read-only pondering arm.<n>A predicted failure in which memory rewriting corrupts the energy signal did not occur.
  arXiv  Detail & Related papers  (2026-07-22T23:34:56Z)
• The Orthogonalized Read Is a Removable Training Scaffold for Recurrent Memory [0.0]
  The effect replicates, but it is not a memory improvement.<n>Training on this task is a long chance plateau followed by a sharp escape.<n>Much of the published gain needs no architecture at all.
  arXiv  Detail & Related papers  (2026-07-02T18:54:23Z)
• CARVE: Content-Aware Recurrent with Value Efficiency for Chunk-Parallel Linear Attention [0.0]
  Recurrent models must forget in order to remember, yet the state of the art decides what to erase without consulting what is stored.<n>This memory-blind gating is one of three coupled defects in the leading delta-rule architecture (GDN-2)<n>We introduce CARVE, which resolves all three problems through one principle: erase only on the key axis.
  arXiv  Detail & Related papers  (2026-06-25T16:16:51Z)
• Control-Plane Placement Shapes Forgetting: An Architectural Study of Agent Memory Across Thirteen System Configurations [1.2763567932588586]
  ForgetEval is a 1000-case templated suite plus a 385-case adversarial layer (132 hand-crafted + 253 LLM-drafted oracle-validated)<n>We observe three placement regimes with partly complementary coverage: deterministic primitives suffice for lexical/temporal categories but fail canonicalization.<n>We expose the trade-off via ForgetEval, a 1000-case templated suite plus a 385-case adversarial layer.
  arXiv  Detail & Related papers  (2026-06-14T16:32:15Z)
• TASR: Training-Free Adaptive Stopping for Iterative Retrieval [12.177557521540082]
  Iterative retrieval-augmented generation agents commonly overspend by continuing to retrieve after the model has converged on an answer.<n>We propose TASR (Training-Free Adaptive Stopping Rule), a one-line predicate that fires when the model repeats its previous-round normalized answer.
  arXiv  Detail & Related papers  (2026-06-11T18:35:14Z)
• $μ$VLA: On Recurrent Memory for Partially Observable Manipulation in VLA Models [44.66763491853812]
  We present a controlled isolation study of recurrence in a strong pretrained VLA backbone.<n>Our formulation augments the transformer with a small set of learnable memory tokens carried across timesteps.<n>We interpret these results as a calibration of the capability envelope of minimal in-backbone recurrence.
  arXiv  Detail & Related papers  (2026-06-10T13:26:40Z)
• Right Makes Might: Aligning Verified Hidden States Empowers RL Reasoning [55.264863369127774]
  Current methods reduce each correct rollout to a single reward bit, ignoring the geometric structure shared among their hidden states.<n>We propose Hidden-Align, an auxiliary loss function that aligns the last-layer hidden states of correct rollouts at the anchor token during RL training, with zero overhead in both training and inference.<n>On eight mathematical reasoning benchmarks, Hidden-Align improves average pass@1 over the DAPO baseline by 3.8, 6.2, and 5.4 percentage points on Qwen3-1.7B, 4B, and 14B respectively, with consistent pass@k gains across all three scales.
  arXiv  Detail & Related papers  (2026-06-02T06:51:15Z)
• Amplifying, Not Learning: Fine-Tuned AI Text Detectors Amplify a Pretrained Direction [51.56484100374058]
  Text detectors amplify a pretrained typicality axis; they do not construct an AI-vs-human boundary.<n>On raw encoders before any task supervision, projecting onto centroid(AI)-centroid(HC3) achieves NYT-vs-HC3 AUROC 0.806/0.944/0.834 across three architectures.<n>On RoBERTa-base, raw projection exceeds fine-tuning; on RoBERTa-base, full fine-tuning reduces discrimination below raw on both fluent-formal populations tested.
  arXiv  Detail & Related papers  (2026-05-20T19:08:38Z)
• Echo: KV-Cache-Free Associative Recall with Spectral Koopman Operators [45.88028371034407]
  KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA)<n>We introduce Echo, a KV-cache-free associative recall architecture built around Spectral Koopman Attention (SKA)
  arXiv  Detail & Related papers  (2026-05-07T22:26:27Z)
• MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval [37.54115468116941]
  This report introduces MemReranker, a reranking model family (0.6B/4B) built on Qwen3-Reranker through multi-stage LLM knowledge distillation.<n>On the memory retrieval benchmark, MemReranker-0.6B substantially outperforms BGE-Reranker and matches open-source 4B/8B models as well as GPT-4o-mini on key metrics.<n>MemReranker-4B further achieves 0.737 MAP, with several metrics on par with Gemini-3-Flash, while maintaining inference latency at only 10-20% of large models.
  arXiv  Detail & Related papers  (2026-05-07T12:33:57Z)
• ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Models [60.14219417402433]
  Existing memory benchmarks for LLM agents evaluate explicit recall of facts, yet overlook implicit memory where experience becomes automated behavior without conscious retrieval.<n>We introduce ImplicitMemBench, the first systematic benchmark evaluating implicit memory through three constructs drawn from standard cognitive-science accounts of non-declarative memory.<n>Our 300-item suite employs a unified Learning/Priming-Interfere-Test protocol with first-attempt scoring.
  arXiv  Detail & Related papers  (2026-04-09T10:26:32Z)
• Predicting Overtakes in Trucks Using CAN Data [51.28632782308621]
  We investigate the detection of truck overtakes from CAN data. Our analysis covers up to 10 seconds before the overtaking event. We observe that the prediction scores of the overtake class tend to increase as we approach the overtake trigger.
  arXiv  Detail & Related papers  (2024-04-08T17:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.