Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models

• URL: http://arxiv.org/abs/2603.14517v1
• Date: Sun, 15 Mar 2026 17:54:43 GMT
• Title: Learning to Forget: Sleep-Inspired Memory Consolidation for Resolving Proactive Interference in Large Language Models
• Abstract summary: Large language models (LLMs) suffer from proactive interference (PI): outdated information in the context window disrupts retrieval of current values.<n>We propose SleepGate, a framework that augments transformer-based LLMs with a learned sleep cycle over the key-value ( KV) cache.<n>SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while all five baselines -- full KV cache, sliding window, H2O, StreamingLLM, and decay-only ablation -- remain below 18%.
• Score: 0.11080037957254413
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) suffer from proactive interference (PI): outdated information in the context window disrupts retrieval of current values. This interference degrades retrieval accuracy log-linearly as stale associations accumulate, a bottleneck that persists regardless of context length and resists prompt-engineering mitigations. Biological brains resolve an analogous challenge through sleep-dependent memory consolidation: synaptic downscaling, selective replay, and targeted forgetting. We propose SleepGate, a biologically inspired framework that augments transformer-based LLMs with a learned sleep cycle over the key-value (KV) cache. SleepGate introduces three mechanisms: (1) a conflict-aware temporal tagger detecting when new entries supersede old ones; (2) a lightweight forgetting gate trained to selectively evict or compress stale cache entries; and (3) a consolidation module that merges surviving entries into compact summaries. These components activate periodically during inference in sleep micro-cycles, governed by an adaptive entropy-based trigger. We formalize a dual-phase training objective jointly optimizing language modeling during the wake phase and post-consolidation retrieval during the sleep phase. Theoretical analysis shows SleepGate reduces the interference horizon from O(n) to O(log n). In experiments with a small-scale transformer (4 layers, 793K parameters), SleepGate achieves 99.5% retrieval accuracy at PI depth 5 and 97.0% at depth 10, while all five baselines -- full KV cache, sliding window, H2O, StreamingLLM, and decay-only ablation -- remain below 18%. Our framework offers an architecture-level solution that prompt engineering cannot address.

Related papers

• NanoSleep: A Parameter-Efficient Hybrid Temporal Convolutional Network for Single-Channel Sleep Stage Classification [0.0]
  We present NanoSleep, a compact hybrid temporal convolutional network for automatic sleep stage classification.<n>NanoSleep provides an effective balance between accuracy and efficiency, making it well suited for wearable devices, home-based sleep monitoring, and resource-constrained clinical applications.
  arXiv  Detail & Related papers  (2026-08-19T06:08:24Z)
• Omni2LoRA: Coherence-Preserving Parametric Memory for Efficient Omni Language Models [65.49950267695267]
  We introduce Omni2LoRA, a framework for efficient parametric memory compression via coherence-preserving context distillation.<n>By converting multimodal memory into a fixed-budget, reusable parameter state, our method drives answer-time multimodal-token load to zero.
  arXiv  Detail & Related papers  (2026-08-10T07:49:10Z)
• Do Language Models Need Sleep? Offline Recurrence for Improved Online Inference [58.46323651532913]
  We study a sleep-like consolidation mechanism in which a model periodically converts recent context into persistent fast weights.<n>During sleep, the model performs $N$ offline recurrent passes over the accumulated context and updates the fast weights in its state-space model.<n>We show that increasing sleep duration $N$ for our models improves performance, with the largest gains on examples that require deeper reasoning.
  arXiv  Detail & Related papers  (2026-05-25T17:55:39Z)
• SCM: Sleep-Consolidated Memory with Algorithmic Forgetting for Large Language Models [0.0]
  We present a memory architecture for large language models that draws on neuroscientific principles to address the absence of persistent, structured, and biologically plausible memory.<n>SCM implements five core components inspired by human memory: a limited-capacity working memory, multi-dimensional importance tagging, offline sleep-stage consolidation with distinct NREM and REM phases, intentional value-based forgetting, and a computational self-model enabling introspection.<n>Across a standardized benchmark suite of eight tests, the prototype achieves perfect recall accuracy over ten-turn conversations while reducing memory noise by 90.9% through adaptive forgetting.
  arXiv  Detail & Related papers  (2026-04-22T15:47:27Z)
• CAWN: Continuous Acoustic Wave Networks for Autoregressive Language Modeling [46.16066322190728]
  We introduce the Continuous Acoustic Wave Network (CAWN), a fully continuous sequence-mixing architecture.<n>Instead of discrete matrix-based attention, CAWN projects hidden states into multi-headed complex-domain phasors.<n>To prevent signal degradation over ultra-long contexts, we introduce a dual-gated Selective Phase Resonance mechanism.
  arXiv  Detail & Related papers  (2026-04-05T20:13:22Z)
• MemDLM: Memory-Enhanced DLM Training [56.40248490616793]
  Diffusion Language Models (DLMs) offer attractive advantages over Auto-Regressive (AR) models.<n>They suffer from a notable train-inference mismatch.<n>We propose MemDLM (Memory-Enhanced DLM), which embeds a simulated denoising process into training.
  arXiv  Detail & Related papers  (2026-03-23T17:39:56Z)
• Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval [59.295767860331004]
  RF-Mem is a familiarity uncertainty-guided dual-path memory retriever.<n>It embeds human-like dual-process recognition into the retriever.<n>It consistently outperforms one-shot retrieval and full-context reasoning under fixed budget and latency constraints.
  arXiv  Detail & Related papers  (2026-03-10T06:31:44Z)
• From Verbatim to Gist: Distilling Pyramidal Multimodal Memory via Semantic Information Bottleneck for Long-Horizon Video Agents [78.30630000529133]
  We propose MM-Mem, a pyramidal multimodal memory architecture grounded in Fuzzy-Trace Theory.<n> MM-Mem memory structures hierarchically into a Sensory Buffer, Episodic Stream, and Symbolic.<n>Experiments confirm the effectiveness of MM-Mem on both offline and streaming tasks.
  arXiv  Detail & Related papers  (2026-03-02T05:12:45Z)
• Rethinking Multi-Condition DiTs: Eliminating Redundant Attention via Position-Alignment and Keyword-Scoping [61.459927600301654]
  Multi-condition control is bottlenecked by the conventional concatenate-and-attend'' strategy.<n>Our analysis reveals that much of this cross-modal interaction is spatially or semantically redundant.<n>We propose Position-aligned and Keyword-scoped Attention (PKA), a highly efficient framework designed to eliminate these redundancies.
  arXiv  Detail & Related papers  (2026-02-06T16:39:10Z)
• SYNAPSE: Empowering LLM Agents with Episodic-Semantic Memory via Spreading Activation [29.545442480332515]
  We introduce Synapse, a unified memory architecture that transcends static rather than pre-computed links.<n>We show that Synapse significantly outperforms state-of-the-art methods in complex temporal and multi-hop reasoning tasks.<n>Our code and data will be made publicly available upon acceptance.
  arXiv  Detail & Related papers  (2026-01-06T06:19:58Z)
• SimpleMem: Efficient Lifelong Memory for LLM Agents [73.74399447715052]
  We introduce SimpleMem, an efficient memory framework based on semantic lossless compression.<n>We propose a three-stage pipeline designed to maximize information density and token utilization.<n> Experiments on benchmark datasets show that our method consistently outperforms baseline approaches in accuracy, retrieval efficiency, and inference cost.
  arXiv  Detail & Related papers  (2026-01-05T21:02:49Z)
• StableSleep: Source-Free Test-Time Adaptation for Sleep Staging with Lightweight Safety Rails [0.0]
  Method is model-agnostic, requires no source data or patient calibration, and is practical for on-device or bedside use.<n>We show consistent gains over a frozen baseline of seconds-level latency and minimal memory, reporting per-stage metrics and Cohen's k.<n>The method is model-agnostic, requires no source data or patient calibration, and is practical for on-device or bedside use.
  arXiv  Detail & Related papers  (2025-09-03T03:42:31Z)
• Sparse-dLLM: Accelerating Diffusion LLMs with Dynamic Cache Eviction [72.27673320976933]
  Diffusion Large Language Models (dLLMs) enable breakthroughs in reasoning and parallel decoding.<n>Current caching techniques accelerate decoding by storing full-layer states, yet impose substantial memory usage.<n>We propose Sparse-dLLM, the first training-free framework integrating dynamic cache eviction with sparse attention.
  arXiv  Detail & Related papers  (2025-08-04T16:14:03Z)
• Training-Free Exponential Context Extension via Cascading KV Cache [49.608367376911694]
  We introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens.<n>Our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens.
  arXiv  Detail & Related papers  (2024-06-24T03:59:17Z)
• L-SeqSleepNet: Whole-cycle Long Sequence Modelling for Automatic Sleep Staging [16.96499618061823]
  L-SeqSleepNet is a new deep learning model that takes into account whole-cycle sleep information for sleep staging. L-SeqSleepNet is able to alleviate the predominance of N2 sleep to bring down errors in other sleep stages.
  arXiv  Detail & Related papers  (2023-01-09T15:44:43Z)
• A Closed-loop Sleep Modulation System with FPGA-Accelerated Deep Learning [1.5569382274788235]
  We develop a sleep modulation system that supports closed-loop operations on a low-power field-programmable gate array (FPGA) device. Deep learning (DL) model is accelerated by a low-power field-programmable gate array (FPGA) device. Model has been validated using a public sleep database containing 81 subjects, achieving a state-of-the-art classification accuracy of 85.8% and a F1-score of 79%.
  arXiv  Detail & Related papers  (2022-11-19T01:47:53Z)
• Neural Network Based Sleep Phases Classification for Resource Constraint Environments [0.0]
  We propose a memory efficient sleep tracking architecture which can work in the embedded environment without needing any cloud or mobile connection. In this study, a novel architecture is proposed that consists of a feature extraction and Artificial Neural Networks based stacking. Over the collected dataset, the proposed classification architecture 20% and 14% better F1 scores than its competitors.
  arXiv  Detail & Related papers  (2021-05-25T11:06:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.