Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory

• URL: http://arxiv.org/abs/2607.27919v1
• Date: Thu, 30 Jul 2026 09:30:03 GMT
• Title: Memory Decoder at Scale: A Pretrained, Parametric Long-Term Memory
• Abstract summary: Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale.<n>We present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens.<n>We find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone.
• Score: 39.928280043928325
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decoder-only language models entangle long-term memory and reasoning in a single parameter set, making it difficult to scale memory capacity independently. Memory Decoder introduces a parametric long-term memory module but only studies it at a relatively small scale. In this work, we present Memory Decoder at Scale, scaling memory models up to 6.9B parameters and pretraining them on 300B tokens. At this data scale, the combined cost of indexing and search makes a standard Faiss pipeline infeasible. We address this bottleneck with a distributed pipeline for Faiss indexing and retrieval, together with sparse, batch-wise loading of kNN distributions. Across model scales, we find that allocating more parameters to memory yields a better parameter-performance tradeoff than scaling the base model alone. On 17 benchmarks, pairing a 6.9B general memory with Pythia-410M raises its average score from 29.86 to 37.34, surpassing Pythia-12B (37.24) with 39% fewer total parameters. For Qwen3 Base models ranging from 0.6B to 14B, 1.7B domain memories improve the average score across the three domains by more than 9 points at every scale. Overall, our results demonstrate that independently scaling pretrained memory offers a more parameter efficient path to improving language model performance.

Related papers

• Memory Grafting: Scaling Language Model Pre-training via Offline Conditional Memory [65.39827296429527]
  Scaling conditional memory offers a promising way to increase language-model capacity.<n>Existing methods such as Engram learn large memory tables from scratch during pre-training.<n>We propose Memory Grafting, a conditional memory scaling method that utilizes frozen hidden states from a grafting model as conditional n-gram memory.
  arXiv  Detail & Related papers  (2026-05-20T09:35:04Z)
• Not All Bits Are Equal: Scale-Dependent Memory Optimization Strategies for Reasoning Models [10.604862875916103]
  4-bit quantization has emerged as a memory-optimal choice for non-reasoning models and zero-shot tasks across scales.<n>We show that this universal prescription fails for reasoning models, where the KV cache rather than model size can dominate memory.<n>We find a scale-dependent trade-off: models with an effective size below 8-bit 4B parameters achieve better accuracy by allocating memory to more weights rather than longer generation.
  arXiv  Detail & Related papers  (2025-10-13T03:14:28Z)
• Pretraining with hierarchical memories: separating long-tail and common knowledge [32.22296691842835]
  We introduce small language models that access large hierarchical parametric memory banks encoding world knowledge.<n>During pretraining and inference, we fetch a small, context-dependent memory block and add it to the model.<n>Our pretraining learns to store long-tail world knowledge in the memory parameters, while the small language model acts as an anchor capturing general reasoning abilities.
  arXiv  Detail & Related papers  (2025-09-29T17:59:50Z)
• Memory Layers at Scale [67.00854080570979]
  This work takes memory layers beyond proof-of-concept, proving their utility at contemporary scale.<n>On downstream tasks, language models augmented with our improved memory layer outperform dense models with more than twice the budget, as well as mixture-of-expert models when matched for both compute and parameters.<n>We provide a fully parallelizable memory layer implementation, demonstrating scaling laws with up to 128B memory parameters, pretrained to 1 trillion tokens, comparing to base models with up to 8B parameters.
  arXiv  Detail & Related papers  (2024-12-12T23:56:57Z)
• QLoRA: Efficient Finetuning of Quantized LLMs [66.58009990713134]
  We present QLoRA, an efficient finetuning approach that reduces memory usage enough to finetune a 65B parameter model on a single 48GB GPU. QLoRA backpropagates through a frozen, 4-bit quantized pretrained language model into Low Rank Adapters(LoRA) Our best model family, which we name Guanaco, outperforms all previous openly released models on the Vicuna benchmark.
  arXiv  Detail & Related papers  (2023-05-23T17:50:33Z)
• A Model or 603 Exemplars: Towards Memory-Efficient Class-Incremental Learning [56.450090618578]
  Class-Incremental Learning (CIL) aims to train a model with limited memory size to meet this requirement. We show that when counting the model size into the total budget and comparing methods with aligned memory size, saving models do not consistently work. We propose a simple yet effective baseline, denoted as MEMO for Memory-efficient Expandable MOdel.
  arXiv  Detail & Related papers  (2022-05-26T08:24:01Z)
• Differentiable Random Access Memory using Lattices [0.0]
  We introduce a differentiable random access memory module with $O(1)$ performance regardless of size. The design stores entries on points of a chosen lattice to calculate nearest neighbours of arbitrary points efficiently by exploiting symmetries.
  arXiv  Detail & Related papers  (2021-07-07T20:55:42Z)
• Memformer: A Memory-Augmented Transformer for Sequence Modeling [55.780849185884996]
  We present Memformer, an efficient neural network for sequence modeling. Our model achieves linear time complexity and constant memory space complexity when processing long sequences.
  arXiv  Detail & Related papers  (2020-10-14T09:03:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.