CoME: An Unlearning-based Approach to Conflict-free Model Editing

• URL: http://arxiv.org/abs/2502.15826v1
• Date: Thu, 20 Feb 2025 04:55:38 GMT
• Title: CoME: An Unlearning-based Approach to Conflict-free Model Editing
• Authors: Dahyun Jung, Jaehyung Seo, Jaewook Lee, Chanjun Park, Heuiseok Lim,
• Abstract summary: Large language models (LLMs) often retain outdated or incorrect information from pre-training, which undermines their reliability.<n>We propose Conflict-free Model Editing (CoME), a novel framework that enhances the accuracy of knowledge updates in LLMs by selectively removing outdated knowledge.
• Score: 8.215201299292033
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) often retain outdated or incorrect information from pre-training, which undermines their reliability. While model editing methods have been developed to address such errors without full re-training, they frequently suffer from knowledge conflicts, where outdated information interferes with new knowledge. In this work, we propose Conflict-free Model Editing (CoME), a novel framework that enhances the accuracy of knowledge updates in LLMs by selectively removing outdated knowledge. CoME leverages unlearning to mitigate knowledge interference, allowing new information to be integrated without compromising relevant linguistic features. Through experiments on GPT-J and LLaMA-3 using Counterfact and ZsRE datasets, we demonstrate that CoME improves both editing accuracy and model reliability when applied to existing editing methods. Our results highlight that the targeted removal of outdated knowledge is crucial for enhancing model editing effectiveness and maintaining the model's generative performance.

Related papers

• K-Edit: Language Model Editing with Contextual Knowledge Awareness [71.73747181407323]
  Knowledge-based model editing enables precise modifications to the weights of large language models. We present K-Edit, an effective approach to generating contextually consistent knowledge edits.
  arXiv  Detail & Related papers  (2025-02-15T01:35:13Z)
• LLMs as Repositories of Factual Knowledge: Limitations and Solutions [1.7764955091415962]
  We study the appropriateness of Large Language Models (LLMs) as repositories of factual knowledge. We evaluate their reliability in responding to time-sensitive factual questions. We propose "ENtity-Aware Fine-tuning" (ENAF) to improve the model's performance.
  arXiv  Detail & Related papers  (2025-01-22T10:16:53Z)
• Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization [48.07144492109635]
  Large language models need to be updated regularly. Model editing is challenging as it might also affect knowledge that is unrelated to the new data. We propose SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization.
  arXiv  Detail & Related papers  (2024-10-03T12:28:13Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors. Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update. We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• In-Context Editing: Learning Knowledge from Self-Induced Distributions [29.10148782152867]
  We introduce Consistent In-Context Editing (ICE) to optimize toward a contextual distribution rather than a one-hot target.<n>ICE enhances the robustness and effectiveness of gradient-based tuning methods, preventing overfitting and preserving the model's integrity.<n>We analyze ICE across four critical aspects of knowledge editing: accuracy, locality, generalization, and linguistic quality, demonstrating its advantages.
  arXiv  Detail & Related papers  (2024-06-17T04:00:04Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)
• A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
  Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches. We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
  arXiv  Detail & Related papers  (2024-01-02T16:54:58Z)
• History Matters: Temporal Knowledge Editing in Large Language Model [42.74144542674756]
  We introduce the task of Temporal Knowledge Editing (TKE) and establish a benchmark AToKe to evaluate current model editing methods. We find that while existing model editing methods are effective at making models remember new knowledge, the edited model catastrophically forgets historical knowledge. To address this gap, we propose a simple and general framework termed Multi-Editing with Time Objective (METO) for enhancing existing editing models.
  arXiv  Detail & Related papers  (2023-12-09T07:51:56Z)
• Large Language Models with Controllable Working Memory [64.71038763708161]
  Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP) What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining. How the model's world knowledge interacts with the factual information presented in the context remains under explored.
  arXiv  Detail & Related papers  (2022-11-09T18:58:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.