Editing as Unlearning: Are Knowledge Editing Methods Strong Baselines for Large Language Model Unlearning?

• URL: http://arxiv.org/abs/2505.19855v1
• Date: Mon, 26 May 2025 11:39:56 GMT
• Title: Editing as Unlearning: Are Knowledge Editing Methods Strong Baselines for Large Language Model Unlearning?
• Authors: Zexi Li, Xiangzhu Wang, William F. Shen, Meghdad Kurmanji, Xinchi Qiu, Dongqi Cai, Chao Wu, Nicholas D. Lane,
• Abstract summary: editing and unlearning seem to be two distinct tasks, we find there is a tight connection between them.<n>We evaluate if knowledge editing techniques are strong baselines for LLM unlearning.<n>We propose practical recipes including self-improvement and query merging to better adapt editing methods for unlearning applications.
• Score: 14.656572343761153
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language Model (LLM) unlearning, i.e., selectively removing information from LLMs, is vital for responsible model deployment. Differently, LLM knowledge editing aims to modify LLM knowledge instead of removing it. Though editing and unlearning seem to be two distinct tasks, we find there is a tight connection between them. In this paper, we conceptualize unlearning as a special case of editing where information is modified to a refusal or "empty set" $\emptyset$ response, signifying its removal. This paper thus investigates if knowledge editing techniques are strong baselines for LLM unlearning. We evaluate state-of-the-art (SOTA) editing methods (e.g., ROME, MEMIT, GRACE, WISE, and AlphaEdit) against existing unlearning approaches on pretrained and finetuned knowledge. Results show certain editing methods, notably WISE and AlphaEdit, are effective unlearning baselines, especially for pretrained knowledge, and excel in generating human-aligned refusal answers. To better adapt editing methods for unlearning applications, we propose practical recipes including self-improvement and query merging. The former leverages the LLM's own in-context learning ability to craft a more human-aligned unlearning target, and the latter enables ROME and MEMIT to perform well in unlearning longer sample sequences. We advocate for the unlearning community to adopt SOTA editing methods as baselines and explore unlearning from an editing perspective for more holistic LLM memory control.

Related papers

• MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modality [55.01380617388064]
  Most existing methods overfit to specific models, causing edited knowledge to be discarded during each update.<n>We introduce MindBridge, a scalable solution inspired by the low coupling between modality processing and LLMs in multi-modal models.<n>MindBridge achieves superior performance even in editing tens of thousands of knowledge entries and can flexibly adapt to different LLMs.
  arXiv  Detail & Related papers  (2025-03-04T15:17:57Z)
• AnyEdit: Edit Any Knowledge Encoded in Language Models [69.30638272162267]
  We propose AnyEdit, a new autoregressive editing paradigm for large language models (LLMs)<n>It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs.<n>It outperforms strong baselines by 21.5% on benchmarks including UnKEBench, AKEW, and our new EditEverything dataset for long-form diverse-formatted knowledge.
  arXiv  Detail & Related papers  (2025-02-08T16:18:37Z)
• Lifelong Knowledge Editing for LLMs with Retrieval-Augmented Continuous Prompt Learning [30.554641380670315]
  We introduce RECIPE, a ContInuous Prompt lEarning method to boost editing efficacy and inference efficiency in lifelong learning.<n> RECIPE first converts knowledge statements into short and informative continuous prompts, prefixed to the LLM's input query embedding.<n>It further integrates the Knowledge Sentinel (KS) that acts as an intermediary to calculate a dynamic threshold.<n>Our retriever and prompt encoder are jointly trained to achieve editing properties, i.e. reliability, generality, and locality.
  arXiv  Detail & Related papers  (2024-05-06T08:52:11Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)
• Editing Conceptual Knowledge for Large Language Models [65.38231526537476]
  This paper pioneers the investigation of editing conceptual knowledge for Large Language Models (LLMs) We construct a novel benchmark dataset ConceptEdit and establish a suite of new metrics for evaluation. experimental results reveal that, although existing editing methods can efficiently modify concept-level definition to some extent, they also have the potential to distort the related instantial knowledge.
  arXiv  Detail & Related papers  (2024-03-10T16:57:10Z)
• Knowledge Graph Enhanced Large Language Model Editing [37.6721061644483]
  Large language models (LLMs) are pivotal in advancing natural language processing (NLP) tasks. Existing editing methods struggle to track and incorporate changes in knowledge associated with edits. We propose a novel model editing method that leverages knowledge graphs for enhancing LLM editing, namely GLAME.
  arXiv  Detail & Related papers  (2024-02-21T07:52:26Z)
• Learning to Edit: Aligning LLMs with Knowledge Editing [101.96620267293731]
  We propose a Learning to Edit (LTE) framework, focusing on teaching large language models to apply updated knowledge into input questions. LTE features a two-phase process: (i) the Alignment Phase, which fine-tunes LLMs on a meticulously curated parallel dataset to make reliable, in-scope edits. We demonstrate LTE's superiority in knowledge editing performance, robustness in both batch and sequential editing, minimal interference on general tasks, and rapid editing speeds.
  arXiv  Detail & Related papers  (2024-02-19T07:45:17Z)
• On the Robustness of Editing Large Language Models [57.477943944826904]
  Large language models (LLMs) have played a pivotal role in building communicative AI, yet they encounter the challenge of efficient updates. This work seeks to understand the strengths and limitations of editing methods, facilitating practical applications of communicative AI.
  arXiv  Detail & Related papers  (2024-02-08T17:06:45Z)
• EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models [45.70959260613425]
  We propose EasyEdit, an easy-to-use knowledge editing framework for Large Language Models. It supports various cutting-edge knowledge editing approaches and can be readily applied to many well-known LLMs. We report the knowledge editing results on LlaMA-2 with EasyEdit, demonstrating that knowledge editing surpasses traditional fine-tuning.
  arXiv  Detail & Related papers  (2023-08-14T16:52:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.