MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs

• URL: http://arxiv.org/abs/2506.07899v1
• Date: Mon, 09 Jun 2025 16:16:42 GMT
• Title: MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs
• Authors: Ke Wang, Yiming Qin, Nikolaos Dimitriadis, Alessandro Favero, Pascal Frossard,
• Abstract summary: Existing methods for lifelong model editing compromise generalization, interfere with past edits, or fail to scale to long editing sequences.<n>We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory.<n>MeMOIR confines each edit to a distinct subset of the memory parameters, minimizing interference among edits.
• Score: 82.34547399693966
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Language models deployed in real-world systems often require post-hoc updates to incorporate new or corrected knowledge. However, editing such models efficiently and reliably - without retraining or forgetting previous information - remains a major challenge. Existing methods for lifelong model editing either compromise generalization, interfere with past edits, or fail to scale to long editing sequences. We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, i.e., a dedicated parameter module, while preserving the core capabilities of the pre-trained model. By sparsifying input activations through sample-dependent masks, MEMOIR confines each edit to a distinct subset of the memory parameters, minimizing interference among edits. At inference, it identifies relevant edits by comparing the sparse activation patterns of new queries to those stored during editing. This enables generalization to rephrased queries by activating only the relevant knowledge while suppressing unnecessary memory activation for unrelated prompts. Experiments on question answering, hallucination correction, and out-of-distribution generalization benchmarks across LLaMA-3 and Mistral demonstrate that MEMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.

Related papers

• InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing [77.47790551485721]
  In-context learning is a promising editing method by comprehending edit information through context encoding.<n>This method is constrained by the limited context window of large language models.<n>We propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts.
  arXiv  Detail & Related papers  (2025-05-28T09:20:18Z)
• UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Large Language Models [18.23723680134397]
  Lifelong learning enables large language models to adapt to evolving information by continually updating their internal knowledge.<n>Model editing stands out as a promising solution for this goal, offering a focused and efficient way to revise a model's internal knowledge.<n>UltraEdit is training-, subject- and memory-free, making it well-suited for ultra-scalable, real-world lifelong model editing.
  arXiv  Detail & Related papers  (2025-05-20T17:59:04Z)
• Constraining Sequential Model Editing with Editing Anchor Compression [40.93064933191375]
  Large language models (LLMs) struggle with hallucinations due to false or outdated knowledge.<n>This paper statistically observes that the parameter matrix after editing exhibits a significant deviation compared to its previous state as the number of edits increases.<n>A framework termed Editing Anchor Compression (EAC) is proposed to constrain the deviation of the parameter matrix during sequential editing.
  arXiv  Detail & Related papers  (2025-02-25T03:56:49Z)
• O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing [0.0]
  Large language models (LLMs) acquire knowledge during pre-training, but over time, this knowledge may become incorrect or outdated, necessitating updates after training. We propose Orthogonal Subspace Editing, O-Edit. This algorithmizes the direction of each knowledge update, minimizing interference between successive updates and reducing the impact of new updates on unrelated knowledge. It can perform thousands of edits on mainstream LLMs, achieving an average performance improvement that is 4.2 times better than existing methods while effectively preserving the model's performance on downstream tasks, all with minimal additional parameter overhead.
  arXiv  Detail & Related papers  (2024-10-15T10:16:45Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors.<n>Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update.<n>We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models [78.22291694903659]
  Large language models (LLMs) need knowledge updates to meet the ever-growing world facts and correct the hallucinated responses.<n>Where the updated knowledge resides in memories is a fundamental question for model editing.<n>We propose WISE to bridge the gap between memories.
  arXiv  Detail & Related papers  (2024-05-23T16:35:52Z)
• The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse [58.0132400208411]
  Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. benchmarking Large Language Models after each edit is impractically time-consuming and resource-intensive. We have utilized GPT-3.5 to develop a new dataset, HardEdit, based on hard cases.
  arXiv  Detail & Related papers  (2024-02-15T01:50:38Z)
• SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering [17.20346072074533]
  Recent model editing is a promising technique for efficiently updating a small amount of knowledge of large language models.<n>We propose a detachable and expandable Subject Word Embedding Altering (SWEA) framework, which finds the editing embeddings through token-level matching.<n>We demonstrate the overall state-of-the-art (SOTA) performance of SWEA$oplus$OS on the CounterFact and zsRE datasets.
  arXiv  Detail & Related papers  (2024-01-31T13:08:45Z)
• Memory-Based Model Editing at Scale [102.28475739907498]
  Existing model editors struggle to accurately model an edit's intended scope. We propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC) SERAC stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed.
  arXiv  Detail & Related papers  (2022-06-13T23:40:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.