History Matters: Temporal Knowledge Editing in Large Language Model
- URL: http://arxiv.org/abs/2312.05497v3
- Date: Thu, 14 Dec 2023 12:06:24 GMT
- Title: History Matters: Temporal Knowledge Editing in Large Language Model
- Authors: Xunjian Yin, Jin Jiang, Liming Yang, Xiaojun Wan
- Abstract summary: We introduce the task of Temporal Knowledge Editing (TKE) and establish a benchmark AToKe to evaluate current model editing methods.
We find that while existing model editing methods are effective at making models remember new knowledge, the edited model catastrophically forgets historical knowledge.
To address this gap, we propose a simple and general framework termed Multi-Editing with Time Objective (METO) for enhancing existing editing models.
- Score: 42.74144542674756
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The imperative task of revising or updating the knowledge stored within large
language models arises from two distinct sources: intrinsic errors inherent in
the model which should be corrected and outdated knowledge due to external
shifts in the real world which should be updated. Prevailing efforts in model
editing conflate these two distinct categories of edits arising from distinct
reasons and directly modify the original knowledge in models into new
knowledge. However, we argue that preserving the model's original knowledge
remains pertinent. Specifically, if a model's knowledge becomes outdated due to
evolving worldly dynamics, it should retain recollection of the historical
knowledge while integrating the newfound knowledge. In this work, we introduce
the task of Temporal Knowledge Editing (TKE) and establish a benchmark AToKe
(Assessment of TempOral Knowledge Editing) to evaluate current model editing
methods. We find that while existing model editing methods are effective at
making models remember new knowledge, the edited model catastrophically forgets
historical knowledge. To address this gap, we propose a simple and general
framework termed Multi-Editing with Time Objective (METO) for enhancing
existing editing models, which edits both historical and new knowledge
concurrently and optimizes the model's prediction for the time of each fact.
Our assessments demonstrate that while AToKe is still difficult, METO maintains
the effectiveness of learning new knowledge and meanwhile substantially
improves the performance of edited models on utilizing historical knowledge.
Related papers
- Outdated Issue Aware Decoding for Reasoning Questions on Edited Knowledge [93.54427119091174]
We propose outDated ISsue aware deCOding to enhance the performance of edited models on reasoning questions.
We capture the difference in the probability distribution between the original and edited models.
We amplify the difference of the token prediction in the edited model to alleviate the outdated issue.
arXiv Detail & Related papers (2024-06-05T03:00:15Z) - Detecting Edited Knowledge in Language Models [5.260519479124422]
Knowledge editing methods (KEs) can update language models' obsolete or inaccurate knowledge learned from pre-training.
Knowing whether a generated output is based on edited knowledge or first-hand knowledge from pre-training can increase users' trust in generative models.
We propose a novel task: detecting edited knowledge in language models.
arXiv Detail & Related papers (2024-05-04T22:02:24Z) - Updating Language Models with Unstructured Facts: Towards Practical
Knowledge Editing [87.35944788684958]
We propose a new benchmark, Unstructured Knowledge Editing (UKE)
UKE evaluates editing performance directly using unstructured texts as knowledge updates, termed unstructured facts.
We conduct extensive experiments on newly built datasets and demonstrate that UKE poses a significant challenge to state-of-the-art knowledge editing methods.
arXiv Detail & Related papers (2024-02-29T07:08:34Z) - A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication.
This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches.
We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
arXiv Detail & Related papers (2024-01-02T16:54:58Z) - Adaptively Integrated Knowledge Distillation and Prediction Uncertainty
for Continual Learning [71.43841235954453]
Current deep learning models often suffer from catastrophic forgetting of old knowledge when continually learning new knowledge.
Existing strategies to alleviate this issue often fix the trade-off between keeping old knowledge (stability) and learning new knowledge (plasticity)
arXiv Detail & Related papers (2023-01-18T05:36:06Z) - Large Language Models with Controllable Working Memory [64.71038763708161]
Large language models (LLMs) have led to a series of breakthroughs in natural language processing (NLP)
What further sets these models apart is the massive amounts of world knowledge they internalize during pretraining.
How the model's world knowledge interacts with the factual information presented in the context remains under explored.
arXiv Detail & Related papers (2022-11-09T18:58:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.