MEMIT-Merge: Addressing MEMIT's Key-Value Conflicts in Same-Subject Batch Editing for LLMs

• URL: http://arxiv.org/abs/2502.07322v3
• Date: Tue, 09 Sep 2025 04:29:04 GMT
• Title: MEMIT-Merge: Addressing MEMIT's Key-Value Conflicts in Same-Subject Batch Editing for LLMs
• Abstract summary: We show that MEMIT's editing efficacy significantly deteriorates when processing batches containing multiple edits sharing the same subject.<n>We propose MEMIT-Merge, an enhanced approach that merges value processes for facts sharing the same subject.
• Score: 37.374258713584496
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As large language models continue to scale up, knowledge editing techniques that modify models' internal knowledge without full retraining have gained significant attention. MEMIT, a prominent batch editing algorithm, stands out for its capability to perform mass knowledge modifications. However, we uncover that MEMIT's editing efficacy significantly deteriorates when processing batches containing multiple edits sharing the same subject. Our analysis reveals this stems from MEMIT's key value modeling framework: identical keys (derived from the shared subject) are forced to represent different values (corresponding to different knowledge), resulting in update conflicts during editing. Addressing this issue, we propose MEMIT-Merge, an enhanced approach that merges value computation processes for facts sharing the same subject, effectively resolving the performance degradation in samesubject batch editing scenarios. Experimental results demonstrate that when MEMIT's edit success rate drops to around 50% at larger batch sizes, MEMIT-Merge maintains a success rate exceeding 90%, showcasing remarkable robustness to subject entity collisions. The code is available at https://github.com/NUSTM/ MEMIT-Merge.

Related papers

• The Anatomy of an Edit: Mechanism-Guided Activation Steering for Knowledge Editing [52.315369633116255]
  Large language models (LLMs) are increasingly used as knowledge bases, but keeping them up to date requires targeted knowledge editing (KE)<n>We take a mechanistic view of KE using neuron-level knowledge attribution (NLKA)<n>Across representative KE methods, we find a consistent pattern: mid-to-late attention predominantly promotes the new target, while attention and FFN modules cooperate to suppress the original fact.<n>Motivated by these findings, we propose MEGA, a MEchanism-Guided Activation steering method that performs attention-residual interventions in attribution-aligned regions.
  arXiv  Detail & Related papers  (2026-03-21T12:40:15Z)
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs [82.34547399693966]
  Existing methods for lifelong model editing compromise generalization, interfere with past edits, or fail to scale to long editing sequences.<n>We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory.<n>MeMOIR confines each edit to a distinct subset of the memory parameters, minimizing interference among edits.
  arXiv  Detail & Related papers  (2025-06-09T16:16:42Z)
• InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing [77.47790551485721]
  In-context learning is a promising editing method by comprehending edit information through context encoding.<n>This method is constrained by the limited context window of large language models.<n>We propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts.
  arXiv  Detail & Related papers  (2025-05-28T09:20:18Z)
• NAMET: Robust Massive Model Editing via Noise-Aware Memory Optimization [7.647276551638579]
  NAMET (Noise-aware Model Editing in Transformers) is a simple yet effective method that introduces noise during memory extraction via a one-line modification to MEMIT.<n>Experiments across six large language models and three datasets demonstrate that NAMET consistently outperforms existing methods when editing thousands of facts.
  arXiv  Detail & Related papers  (2025-05-17T07:00:02Z)
• AnyEdit: Edit Any Knowledge Encoded in Language Models [69.30638272162267]
  We propose AnyEdit, a new autoregressive editing paradigm for large language models (LLMs) It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. It outperforms strong baselines by 21.5% on benchmarks including UnKEBench, AKEW, and our new EditEverything dataset for long-form diverse-formatted knowledge.
  arXiv  Detail & Related papers  (2025-02-08T16:18:37Z)
• Related Knowledge Perturbation Matters: Rethinking Multiple Pieces of Knowledge Editing in Same-Subject [49.559994791305535]
  Current state-of-the-art editing methods struggle when tasked with editing multiple related knowledge pieces for the same subject.<n>We introduce the $textS2textRKE$(Same-Subject Related Knowledge Editing) benchmark.<n>Our experiments reveal that only mainstream locate-then-edit methods, such as ROME and MEMIT, exhibit "related knowledge perturbation"
  arXiv  Detail & Related papers  (2025-02-08T04:47:17Z)
• ComprehendEdit: A Comprehensive Dataset and Evaluation Framework for Multimodal Knowledge Editing [27.034072044001736]
  Large multimodal language models (MLLMs) have revolutionized natural language processing and visual understanding. Current knowledge editing evaluations are limited in scope and potentially biased. We introduce ComprehendEdit, a comprehensive benchmark comprising eight diverse tasks from multiple datasets.
  arXiv  Detail & Related papers  (2024-12-17T11:41:49Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors. Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update. We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• A Unified Framework for Model Editing [2.569159339315845]
  ROME and MEMIT are largely believed to be two different model editing algorithms. We unify these two algorithms under a single conceptual umbrella, optimizing for the same goal. We show that both algorithms are equivalent in terms of their optimization objective, their abilities, their model editing performance and their limitations.
  arXiv  Detail & Related papers  (2024-03-21T08:54:24Z)
• Navigating the Dual Facets: A Comprehensive Evaluation of Sequential Memory Editing in Large Language Models [37.91572705169647]
  Two mainstream ME methods exist: parameter-modifying ME and parameter-preserving ME. This study explores how ME affects a wide range of fundamental capabilities of Large Language Models (LLMs) under sequential editing.
  arXiv  Detail & Related papers  (2024-02-16T23:08:55Z)
• The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse [58.0132400208411]
  Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. benchmarking Large Language Models after each edit is impractically time-consuming and resource-intensive. We have utilized GPT-3.5 to develop a new dataset, HardEdit, based on hard cases.
  arXiv  Detail & Related papers  (2024-02-15T01:50:38Z)
• Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue [122.20016030723043]
  We evaluate the side effects of model editing on large language models (LLMs) Our analysis reveals that the side effects are caused by model editing altering the original model weights excessively. To mitigate this, a method named RECT is proposed to regularize the edit update weights.
  arXiv  Detail & Related papers  (2024-01-09T18:03:15Z)
• Memory-Based Model Editing at Scale [102.28475739907498]
  Existing model editors struggle to accurately model an edit's intended scope. We propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC) SERAC stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed.
  arXiv  Detail & Related papers  (2022-06-13T23:40:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.