A Parametric Memory Head for Continual Generative Retrieval

• URL: http://arxiv.org/abs/2604.23388v1
• Date: Sat, 25 Apr 2026 17:38:51 GMT
• Title: A Parametric Memory Head for Continual Generative Retrieval
• Abstract summary: Generative information retrieval (GenIR) consolidates retrieval into a single neural model that decodes document identifiers (docids) directly from queries.<n>We show that sequential adaptation improves retrieval on newly added documents but substantially degrades performance on earlier slices.<n>We propose post-adaptation memory tuning (PAMT), a memory-only stabilization stage that augments an adapted model with a modular parametric memory head.
• Score: 52.66674234249913
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative information retrieval (GenIR) consolidates retrieval into a single neural model that decodes document identifiers (docids) directly from queries. While this model-as-index paradigm offers architectural simplicity, it is poorly suited to dynamic document collections. Unlike modular systems, where indexes are easily updated, GenIR's knowledge is parametrically encoded in its weights; consequently, standard adaptation methods such as full and parameter-efficient fine-tuning can induce catastrophic forgetting. We show that sequential adaptation improves retrieval on newly added documents but substantially degrades performance on earlier slices, exposing a pronounced stability-plasticity trade-off. To address this, we propose post-adaptation memory tuning (PAMT), a memory-only stabilization stage that augments an adapted model with a modular parametric memory head (PMH). PAMT freezes the backbone and attaches a product-key memory with fixed addressing. During prefix-trie constrained decoding, decoder hidden states sparsely query PMH to produce residual corrections in hidden space; these corrections are mapped to score adjustments via the frozen output embedding matrix, computed only over trie-valid tokens. This guides docid generation while keeping routing and backbone parameters fixed. To limit cross-slice interference, PAMT updates only a fixed budget of memory values selected using decoding-time access statistics, prioritizing entries frequently activated by the current slice and rarely used in prior sessions. Experiments on MS MARCO and Natural Questions under sequential, disjoint corpus increments show that PAMT substantially improves retention on earlier slices with minimal impact on retrieval performance for newly added documents, while modifying only a sparse subset of memory values per session.

Related papers

• Embedding Surgery: Localized Updates for Adaptive Ranking Correction in Dense Retrieval [7.977161233209229]
  We introduce emphembedding surgery, a lightweight approach for adaptive ranking correction in dense retrieval.<n>The method applies localized, minimal updates to selected document embeddings at query time, guided by editorial feedback, user interactions, or pseudo-labels from large language models.<n>We integrate embedding surgery into standard dense retrieval pipelines and evaluate it on TREC Deep Learning, TREC Robust, TREC CAsT, and MS MARCO benchmarks.
  arXiv  Detail & Related papers  (2026-09-04T13:06:26Z)
• InjecMEM: Memory Injection Attack on LLM Agent Memory Systems [31.104106193922956]
  InjecMEM is a novel memory injection attack paradigm.<n>We craft the injection with a retriever-agnostic anchor and an adversarial command.<n>We evaluate it across multiple memory systems and backbone models.
  arXiv  Detail & Related papers  (2026-08-24T16:37:50Z)
• Dual-Grained Agent Memory and Shapley Context Attribution for Multimodal Agentic Learner [55.80248281761638]
  multimodal large language models (MLLMs) deliver impressive perception yet still falter on scientific and mathematical reasoning.<n>We propose DG-Mem, a dual-grained agentic memory framework that augments a frozen MLLM with a non- externally stored memory built once from training-time rollouts and consulted read-only at test time.
  arXiv  Detail & Related papers  (2026-08-24T13:56:01Z)
• Harness the Memory: A Holistic Evaluation of Memory Substrates in Memory Agents [97.19033877543522]
  We present a controlled harness evaluation of memory substrates for memory-augmented agents.<n>We instrument 26 performance and efficiency metrics under a unified harness.<n>These findings motivate substrate routing as a necessary component of adaptive agent memory systems.
  arXiv  Detail & Related papers  (2026-08-15T03:29:48Z)
• InduceKV: Fixed-Footprint Continual Adaptation of Multimodal LLMs via Inducing KV Memories [6.141047838371132]
  InduceKV is a retrieval-based method that stores each selected training prefix as an attention-ready memory entry.<n>We show that InduceKV consistently improves over PEFT, MoE, replay, and prompt-retrieval baselines under matched memory budgets.
  arXiv  Detail & Related papers  (2026-07-02T10:45:28Z)
• Sparse Prefix Caching for Hybrid and Recurrent LLM Serving [0.7284556903703034]
  We formalize sparse prefix caching as checkpoint placement under a distribution over overlap depths.<n>For use cases where requests share a non-trivial prefix, we show that our method consistently improves the frontier traced by standard asymmetry on real-world data.<n>It preserves exact outputs, does not change the recurrent computation itself or require new recurrent update kernels.
  arXiv  Detail & Related papers  (2026-04-17T09:24:58Z)
• Improving Sparse Memory Finetuning [6.491055956716639]
  Large Language Models (LLMs) are typically static after training, yet real-world applications require continual adaptation to new knowledge without degrading existing capabilities.<n>Standard approaches to updating models, like full finetuning or parameter-efficient methods, face a fundamental trade-off: catastrophic forgetting.<n>Sparse Memory Finetuning (SMF) offers a promising alternative by localizing updates to a small subset of parameters in explicit memory layers.
  arXiv  Detail & Related papers  (2026-04-06T23:16:10Z)
• TS-Memory: Plug-and-Play Memory for Time Series Foundation Models [63.21390142212087]
  Time Series Foundation Models (TSFMs) achieve strong zero-shot forecasting through large-scale pre-training.<n>Existing solutions face a trade-off: Parametric Adaptation can cause catastrophic forgetting, while Non-Parametric Retrieval improves forecasts but incurs high latency due to datastore search.<n>We propose Parametric Memory Distillation and implement it as TS-Memory, a lightweight memory adapter that augments frozen TSFMs.
  arXiv  Detail & Related papers  (2026-02-12T04:16:19Z)
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs [76.28901550926021]
  Existing methods for lifelong model editing compromise generalization, interfere with past edits, or fail to scale to long editing sequences.<n>We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, while preserving the core capabilities of the pre-trained model.<n>MeMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.
  arXiv  Detail & Related papers  (2025-06-09T16:16:42Z)
• Adaptive Additive Parameter Updates of Vision Transformers for Few-Shot Continual Learning [0.0]
  Few-shot class incremental learning (FSCIL) addresses this by first training a model on a robust dataset of base classes and then incrementally adapting it in successive sessions.<n>This approach is prone to overfitting on the limited new data, which can compromise overall performance and exacerbate forgetting.<n>We propose a novel FSCIL framework that leverages a frozen Vision Transformer (ViT) backbone augmented with parameter-efficient additive updates.
  arXiv  Detail & Related papers  (2025-04-11T21:17:30Z)
• DSI++: Updating Transformer Memory with New Documents [95.70264288158766]
  We introduce DSI++, a continual learning challenge for DSI to incrementally index new documents. We show that continual indexing of new documents leads to considerable forgetting of previously indexed documents. We introduce a generative memory to sample pseudo-queries for documents and supplement them during continual indexing to prevent forgetting for the retrieval task.
  arXiv  Detail & Related papers  (2022-12-19T18:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.