MoNe: Modular Neural Memory for Efficient Long Context Inference

• URL: http://arxiv.org/abs/2608.17616v1
• Date: Tue, 18 Aug 2026 10:28:23 GMT
• Title: MoNe: Modular Neural Memory for Efficient Long Context Inference
• Abstract summary: MoNe is a lightweight modular neural memory that attaches to any frozen pretrained Transformer.<n>MoNe reads context in fixed-size segments via test-time learning of fast-weight neural memory networks.<n>MoNe generates keys and values from the query tokens alone, with no context tokens re-read.
• Score: 23.94676543840457
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present MoNe, a lightweight modular neural memory that attaches to any frozen pretrained Transformer to enable long-context inference without retraining. MoNe reads context in fixed-size segments via test-time learning of fast-weight neural memory networks with layer-localized gradient updates; at inference, the memory generates keys and values from the query tokens alone, with no context tokens re-read. This two-phase design decouples inference cost from context length, achieving $O(N)$ preprocessing and $O(1)$ query cost with peak GPU memory that does not grow with $N$. At 128K tokens, MoNe reduces both compute and peak GPU memory by approximately 80% compared to ICL with only 6.4% parameter overhead. MoNe generalizes to context lengths far beyond the backbone's native window, achieving strong performance on needle-in-a-haystack and word extraction benchmarks from RULER, where ICL degrades sharply.

Related papers

• Maglev: Sliding Recurrent Memory [8.788753986099902]
  ours is a recurrent Transformer architecture with fixed-size memory that generalizes sliding-window attention.<n>We train ours with a memory consistency loss that aligns $m_t$ with $m'_t$, allowing inference to use $P$ alone.
  arXiv  Detail & Related papers  (2026-08-03T20:40:49Z)
• DMF: A Deterministic Memory Framework for Conversational AI Agents [0.0]
  We present the Deterministic Memory Framework (DMF), a CPU-first approach that replaces generative memory compression with a fully deterministic pipeline.<n>DMF assigns each conversational interaction a Survival Score $$ computed from deterministic content signals, conversational cues, and structured provenance, combined through a logistic projection.<n>Experiments are conducted on a purpose-built benchmark using the LoCoMo and LongMemEval datasets.
  arXiv  Detail & Related papers  (2026-06-02T10:41:28Z)
• Dynamic Long Context Reasoning over Compressed Memory via End-to-End Reinforcement Learning [47.87361916374891]
  We propose a framework for efficient long-context inference based on chunk-wise compression and selective memory recall.<n>The framework segments long inputs into chunks and encodes each chunk into compressed memory representations using a learned compressor.<n>It achieves up to a 2 times reduction in peak GPU memory usage and a 6 times inference speedup over MemAgent.
  arXiv  Detail & Related papers  (2026-02-09T08:33:11Z)
• Out of the Memory Barrier: A Highly Memory Efficient Training System for LLMs with Million-Token Contexts [68.79341332280062]
  Training Large Language Models (LLMs) on long contexts is severely constrained by prohibitive GPU memory overhead, not training time.<n>We introduce OOMB, a highly memory-efficient training system that directly confronts this barrier.<n>Our approach employs a chunk-recurrent training framework with on-the-fly activation recomputation, which maintains a constant activation memory footprint.
  arXiv  Detail & Related papers  (2026-02-02T13:52:40Z)
• ILRe: Intermediate Layer Retrieval for Context Compression in Causal Language Models [4.951427498576812]
  We introduce a novel context compression pipeline called Intermediate Layer Retrieval (ILRe)<n>ILRe encodes context by streaming chunked prefill only up to that layer, and recalls tokens by the attention scores between the input query and full key cache in that specified layer.<n>Without additional post training or operator development, ILRe can process a single $1M$ tokens request in less than half a minute.
  arXiv  Detail & Related papers  (2025-08-25T10:59:02Z)
• Cut Your Losses in Large-Vocabulary Language Models [102.6981011879656]
  We propose Cut Cross-Entropy (CCE), a method that computes the cross-entropy loss without materializing the logits for all tokens into global memory.<n>CCE reduces the memory footprint of the loss from 24 GB to 1 MB, and the total training-time memory consumption of the head from 28 GB to 1 GB.
  arXiv  Detail & Related papers  (2024-11-13T20:30:15Z)
• Training-Free Exponential Context Extension via Cascading KV Cache [49.608367376911694]
  We introduce a novel mechanism that leverages cascading sub-cache buffers to selectively retain the most relevant tokens.<n>Our method reduces prefill stage latency by a factor of 6.8 when compared to flash attention on 1M tokens.
  arXiv  Detail & Related papers  (2024-06-24T03:59:17Z)
• Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs [61.40047491337793]
  We present Hierarchical cOntext MERging (HOMER), a new training-free scheme designed to overcome the limitations of large language models. HomeR uses a divide-and-conquer algorithm, dividing long inputs into manageable chunks. A token reduction technique precedes each merging, ensuring memory usage efficiency.
  arXiv  Detail & Related papers  (2024-04-16T06:34:08Z)
• LLoCO: Learning Long Contexts Offline [63.3458260335454]
  We propose LLoCO, a novel approach to processing long contexts. LLoCO learns contexts offline through context compression and in-domain parameter-efficient finetuning with LoRA. Our approach extends the effective context window of a 4k token LLaMA2-7B model to handle up to 128k tokens.
  arXiv  Detail & Related papers  (2024-04-11T17:57:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.