Model Merging for Knowledge Editing

• URL: http://arxiv.org/abs/2506.12384v1
• Date: Sat, 14 Jun 2025 07:42:39 GMT
• Title: Model Merging for Knowledge Editing
• Authors: Zichuan Fu, Xian Wu, Guojing Li, Yingying Zhang, Yefeng Zheng, Tianshi Ming, Yejing Wang, Wanyu Wang, Xiangyu Zhao,
• Abstract summary: Large Language Models (LLMs) require continuous updates to maintain accurate and current knowledge as the world evolves.<n>Existing knowledge editing approaches offer various solutions for knowledge updating, but they often struggle with sequential editing scenarios.<n>This paper proposes a two-stage framework combining robust supervised fine-tuning (R-SFT) with model merging for knowledge editing.
• Score: 53.799891745131724
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) require continuous updates to maintain accurate and current knowledge as the world evolves. While existing knowledge editing approaches offer various solutions for knowledge updating, they often struggle with sequential editing scenarios and harm the general capabilities of the model, thereby significantly hampering their practical applicability. This paper proposes a two-stage framework combining robust supervised fine-tuning (R-SFT) with model merging for knowledge editing. Our method first fine-tunes the LLM to internalize new knowledge fully, then merges the fine-tuned model with the original foundation model to preserve newly acquired knowledge and general capabilities. Experimental results demonstrate that our approach significantly outperforms existing methods in sequential editing while better preserving the original performance of the model, all without requiring any architectural changes. Code is available at: https://github.com/Applied-Machine-Learning-Lab/MM4KE.

Related papers

• Retention analysis of edited knowledge after fine-tuning [5.440397659472036]
  Large language models (LLMs) store vast amounts of knowledge, which often requires updates to correct factual errors, incorporate newly acquired information, or adapt model behavior.<n>Model editing methods have emerged as efficient solutions for such updates, offering localized and precise knowledge modification at significantly lower computational cost than continual training.<n>However, the effect of fine-tuning on previously edited knowledge remains poorly understood.
  arXiv  Detail & Related papers  (2025-07-14T15:51:19Z)
• One for All: Update Parameterized Knowledge Across Multiple Models [35.137065486616805]
  Large language models (LLMs) encode vast world knowledge but struggle to stay up-to-date, often leading to errors and hallucinations.<n> Knowledge editing offers an efficient alternative to retraining, enabling targeted modifications by updating specific model parameters.<n>We propose OnceEdit, a novel ensemble-based approach that employs a plug-in model as the editing module.
  arXiv  Detail & Related papers  (2025-06-01T03:48:54Z)
• MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modality [55.01380617388064]
  Most existing methods overfit to specific models, causing edited knowledge to be discarded during each update.<n>We introduce MindBridge, a scalable solution inspired by the low coupling between modality processing and LLMs in multi-modal models.<n>MindBridge achieves superior performance even in editing tens of thousands of knowledge entries and can flexibly adapt to different LLMs.
  arXiv  Detail & Related papers  (2025-03-04T15:17:57Z)
• GeoEdit: Geometric Knowledge Editing for Large Language Models [52.37408324849593]
  Regular updates are essential for maintaining up-to-date knowledge in large language models (LLMs)<n>We propose a novel framework called Geometric Knowledge Editing (GeoEdit)<n>GeoEdit distinguishes between neurons associated with new knowledge updates and those related to general knowledge perturbations.<n>For the remaining neurons, we integrate both old and new knowledge for aligned directions and apply a "forget-then-learn" editing strategy for opposite directions.
  arXiv  Detail & Related papers  (2025-02-27T10:27:48Z)
• CoME: An Unlearning-based Approach to Conflict-free Model Editing [8.215201299292033]
  Large language models (LLMs) often retain outdated or incorrect information from pre-training, which undermines their reliability.<n>We propose Conflict-free Model Editing (CoME), a novel framework that enhances the accuracy of knowledge updates in LLMs by selectively removing outdated knowledge.
  arXiv  Detail & Related papers  (2025-02-20T04:55:38Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors.<n>Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update.<n>We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors [30.831866499812925]
  MEMoE is a model editing adapter utilizing a Mixture of Experts (MoE) architecture with a knowledge anchor routing strategy. We show the superiority of our approach over both batch editing and sequential batch editing tasks.
  arXiv  Detail & Related papers  (2024-05-29T13:49:44Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)
• A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
  Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches. We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
  arXiv  Detail & Related papers  (2024-01-02T16:54:58Z)
• History Matters: Temporal Knowledge Editing in Large Language Model [42.74144542674756]
  We introduce the task of Temporal Knowledge Editing (TKE) and establish a benchmark AToKe to evaluate current model editing methods. We find that while existing model editing methods are effective at making models remember new knowledge, the edited model catastrophically forgets historical knowledge. To address this gap, we propose a simple and general framework termed Multi-Editing with Time Objective (METO) for enhancing existing editing models.
  arXiv  Detail & Related papers  (2023-12-09T07:51:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.