LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models
- URL: http://arxiv.org/abs/2406.20030v1
- Date: Fri, 28 Jun 2024 16:17:41 GMT
- Title: LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models
- Authors: Renzhi Wang, Piji Li,
- Abstract summary: Large language models (LLMs) require continual knowledge updates to stay abreast of the ever-changing world facts.
We introduce LEMoE, an advanced Mixture of Experts (MoE) adaptor for lifelong model editing.
- Score: 30.831866499812925
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Large language models (LLMs) require continual knowledge updates to stay abreast of the ever-changing world facts, prompting the formulation of lifelong model editing task. While recent years have witnessed the development of various techniques for single and batch editing, these methods either fail to apply or perform sub-optimally when faced with lifelong editing. In this paper, we introduce LEMoE, an advanced Mixture of Experts (MoE) adaptor for lifelong model editing. We first analyze the factors influencing the effectiveness of conventional MoE adaptor in lifelong editing, including catastrophic forgetting, inconsistent routing and order sensitivity. Based on these insights, we propose a tailored module insertion method to achieve lifelong editing, incorporating a novel KV anchor routing to enhance routing consistency between training and inference stage, along with a concise yet effective clustering-based editing order planning. Experimental results demonstrate the effectiveness of our method in lifelong editing, surpassing previous model editing techniques while maintaining outstanding performance in batch editing task. Our code will be available.
Related papers
- MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors [30.831866499812925]
MEMoE is a model editing adapter utilizing a Mixture of Experts (MoE) architecture with a knowledge anchor routing strategy.
We show the superiority of our approach over both batch editing and sequential batch editing tasks.
arXiv Detail & Related papers (2024-05-29T13:49:44Z) - Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model Editing with Llama-3 [2.569159339315845]
This study presents a targeted model editing analysis focused on the latest large language model, Llama-3.
We identify the most effective layers for targeted edits through an evaluation that encompasses up to 4096 edits.
arXiv Detail & Related papers (2024-05-01T17:50:37Z) - Consecutive Model Editing with Batch alongside HooK Layers [59.673084839708224]
COMEBA-HK is a model editing method that is both consecutive and batch-supportive.
It is memory-friendly as it only needs a small amount of it to store several hook layers with updated weights.
arXiv Detail & Related papers (2024-03-08T14:07:44Z) - InstructEdit: Instruction-based Knowledge Editing for Large Language Models [39.2147118489123]
We develop an instruction-based editing technique, termed InstructEdit, which facilitates the editor's adaptation to various task performances simultaneously using simple instructions.
Experiments involving holdout unseen task illustrate that InstructEdit consistently surpass previous strong baselines.
arXiv Detail & Related papers (2024-02-25T15:46:33Z) - Learning to Edit: Aligning LLMs with Knowledge Editing [101.96620267293731]
We propose a Learning to Edit (LTE) framework, focusing on teaching large language models to apply updated knowledge into input questions.
LTE features a two-phase process: (i) the Alignment Phase, which fine-tunes LLMs on a meticulously curated parallel dataset to make reliable, in-scope edits.
We demonstrate LTE's superiority in knowledge editing performance, robustness in both batch and sequential editing, minimal interference on general tasks, and rapid editing speeds.
arXiv Detail & Related papers (2024-02-19T07:45:17Z) - The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse [58.0132400208411]
Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks.
benchmarking Large Language Models after each edit is impractically time-consuming and resource-intensive.
We have utilized GPT-3.5 to develop a new dataset, HardEdit, based on hard cases.
arXiv Detail & Related papers (2024-02-15T01:50:38Z) - Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue [122.20016030723043]
Model editing is a technique that edits large language models (LLMs) with updated knowledge to alleviate hallucinations without resource-intensive retraining.
Current model editing methods can effectively modify a model's behavior within a specific area of interest.
They often overlook the potential unintended side effects on the general abilities of LLMs.
arXiv Detail & Related papers (2024-01-09T18:03:15Z) - DUnE: Dataset for Unified Editing [3.7346004746366384]
We introduce DUnE-an editing benchmark where edits are natural language sentences.
We show that retrieval-augmented language modeling can outperform specialized editing techniques.
arXiv Detail & Related papers (2023-11-27T18:56:14Z) - Emu Edit: Precise Image Editing via Recognition and Generation Tasks [62.95717180730946]
We present Emu Edit, a multi-task image editing model which sets state-of-the-art results in instruction-based image editing.
We train it to multi-task across an unprecedented range of tasks, such as region-based editing, free-form editing, and Computer Vision tasks.
We show that Emu Edit can generalize to new tasks, such as image inpainting, super-resolution, and compositions of editing tasks, with just a few labeled examples.
arXiv Detail & Related papers (2023-11-16T18:55:58Z) - Editing Large Language Models: Problems, Methods, and Opportunities [51.903537096207]
This paper embarks on a deep exploration of the problems, methods, and opportunities related to model editing for LLMs.
We provide an exhaustive overview of the task definition and challenges associated with model editing, along with an in-depth empirical analysis of the most progressive methods currently at our disposal.
Our objective is to provide valuable insights into the effectiveness and feasibility of each editing technique, thereby assisting the community in making informed decisions on the selection of the most appropriate method for a specific task or context.
arXiv Detail & Related papers (2023-05-22T16:00:00Z) - CoditT5: Pretraining for Source Code and Natural Language Editing [34.77621217370665]
CoditT5 is a large language model for software-related editing tasks that is pretrained on large amounts of source code and natural language comments.
We fine-tune it on various downstream editing tasks, including comment updating, bug fixing, and automated code review.
arXiv Detail & Related papers (2022-08-10T16:59:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.