WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing
- URL: http://arxiv.org/abs/2402.10987v2
- Date: Wed, 5 Jun 2024 07:44:30 GMT
- Title: WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing
- Authors: Chenhui Hu, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao,
- Abstract summary: This study reveals a performance degradation encountered by knowledge editing in lifelong editing.
We introduce a knowledge editing approach named Wise-Layer Knowledge Editor (WilKE)
WilKE selects editing layer based on the pattern matching degree of editing knowledge across different layers in language models.
- Score: 19.357663224043534
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Knowledge editing aims to rectify inaccuracies in large language models (LLMs) without costly retraining for outdated or erroneous knowledge. However, current knowledge editing methods primarily focus on single editing, failing to meet the requirements for lifelong editing. This study reveals a performance degradation encountered by knowledge editing in lifelong editing, characterized by toxicity buildup and toxicity flash, with the primary cause identified as pattern unmatch. We introduce a knowledge editing approach named Wise-Layer Knowledge Editor (WilKE), which selects editing layer based on the pattern matching degree of editing knowledge across different layers in language models. Experimental results demonstrate that, in lifelong editing, WilKE exhibits an average improvement of 46.2% and 67.8% on editing GPT2-XL and GPT-J relative to state-of-the-art knowledge editing methods.
Related papers
- Editing the Mind of Giants: An In-Depth Exploration of Pitfalls of Knowledge Editing in Large Language Models [26.516571783335824]
Recent studies have identified concerning side effects, such as knowledge distortion and the deterioration of general abilities, that have emerged after editing.
This survey presents a comprehensive study of these side effects, providing a unified view of the challenges associated with knowledge editing in Large Language Models.
arXiv Detail & Related papers (2024-06-03T15:28:21Z) - Editing Factual Knowledge and Explanatory Ability of Medical Large Language Models [89.13883089162951]
Model editing aims to precisely alter the behaviors of large language models (LLMs) in relation to specific knowledge.
This approach has proven effective in addressing issues of hallucination and outdated information in LLMs.
However, the potential of using model editing to modify knowledge in the medical field remains largely unexplored.
arXiv Detail & Related papers (2024-02-28T06:40:57Z) - InstructEdit: Instruction-based Knowledge Editing for Large Language Models [39.2147118489123]
We develop an instruction-based editing technique, termed InstructEdit, which facilitates the editor's adaptation to various task performances simultaneously using simple instructions.
Experiments involving holdout unseen task illustrate that InstructEdit consistently surpass previous strong baselines.
arXiv Detail & Related papers (2024-02-25T15:46:33Z) - Knowledge Graph Enhanced Large Language Model Editing [37.6721061644483]
Large language models (LLMs) are pivotal in advancing natural language processing (NLP) tasks.
Existing editing methods struggle to track and incorporate changes in knowledge associated with edits.
We propose a novel model editing method that leverages knowledge graphs for enhancing LLM editing, namely GLAME.
arXiv Detail & Related papers (2024-02-21T07:52:26Z) - EVEDIT: Event-based Knowledge Editing with Deductive Editing Boundaries [69.72012539060731]
We introduce a theoretical framework for efficient knowledge editing (KE) in large language models (LLMs)
We propose a novel task of event-based knowledge editing that pairs facts with event descriptions.
We empirically demonstrate the superiority of event-based editing over the existing setting on resolving uncertainty in edited models.
arXiv Detail & Related papers (2024-02-17T16:34:50Z) - Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue [122.20016030723043]
Model editing is a technique that edits large language models (LLMs) with updated knowledge to alleviate hallucinations without resource-intensive retraining.
Current model editing methods can effectively modify a model's behavior within a specific area of interest.
They often overlook the potential unintended side effects on the general abilities of LLMs.
arXiv Detail & Related papers (2024-01-09T18:03:15Z) - DUnE: Dataset for Unified Editing [3.7346004746366384]
We introduce DUnE-an editing benchmark where edits are natural language sentences.
We show that retrieval-augmented language modeling can outperform specialized editing techniques.
arXiv Detail & Related papers (2023-11-27T18:56:14Z) - EasyEdit: An Easy-to-use Knowledge Editing Framework for Large Language Models [45.70959260613425]
We propose EasyEdit, an easy-to-use knowledge editing framework for Large Language Models.
It supports various cutting-edge knowledge editing approaches and can be readily applied to many well-known LLMs.
We report the knowledge editing results on LlaMA-2 with EasyEdit, demonstrating that knowledge editing surpasses traditional fine-tuning.
arXiv Detail & Related papers (2023-08-14T16:52:42Z) - Does Localization Inform Editing? Surprising Differences in
Causality-Based Localization vs. Knowledge Editing in Language Models [68.03946716358335]
We find that we can change how a fact is stored in a model by editing weights that are in a different location than where existing methods suggest that the fact is stored.
This is surprising because we would expect that localizing facts to specific model parameters would tell us where to manipulate knowledge in models.
Our results suggest, counterintuitively, that better mechanistic understanding of how pretrained language models work may not always translate to insights about how to best change their behavior.
arXiv Detail & Related papers (2023-01-10T21:26:08Z) - Learning Structural Edits via Incremental Tree Transformations [102.64394890816178]
We present a generic model for incremental editing of structured data (i.e., "structural edits")
Our editor learns to iteratively generate tree edits (e.g., deleting or adding a subtree) and applies them to the partially edited data.
We evaluate our proposed editor on two source code edit datasets, where results show that, with the proposed edit encoder, our editor significantly improves accuracy over previous approaches.
arXiv Detail & Related papers (2021-01-28T16:11:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.