DUnE: Dataset for Unified Editing
- URL: http://arxiv.org/abs/2311.16087v1
- Date: Mon, 27 Nov 2023 18:56:14 GMT
- Title: DUnE: Dataset for Unified Editing
- Authors: Afra Feyza Aky\"urek, Eric Pan, Garry Kuwanto, Derry Wijaya
- Abstract summary: We introduce DUnE-an editing benchmark where edits are natural language sentences.
We show that retrieval-augmented language modeling can outperform specialized editing techniques.
- Score: 3.7346004746366384
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Even the most advanced language models remain susceptible to errors
necessitating to modify these models without initiating a comprehensive
retraining process. Model editing refers to the modification of a model's
knowledge or representations in a manner that produces the desired outcomes.
Prior research primarily centered around editing factual data e.g. "Messi plays
for Inter Miami" confining the definition of an edit to a knowledge triplet
i.e. (subject, object, relation). However, as the applications of language
models expand, so do the diverse ways in which we wish to edit and refine their
outputs. In this study, we broaden the scope of the editing problem to include
an array of editing cases such as debiasing and rectifying reasoning errors and
define an edit as any natural language expression that solicits a change in the
model's outputs. We are introducing DUnE-an editing benchmark where edits are
natural language sentences and propose that DUnE presents a challenging yet
relevant task. To substantiate this claim, we conduct an extensive series of
experiments testing various editing approaches to address DUnE, demonstrating
their respective strengths and weaknesses. We show that retrieval-augmented
language modeling can outperform specialized editing techniques and neither set
of approaches has fully solved the generalized editing problem covered by our
benchmark.
Related papers
- Should We Really Edit Language Models? On the Evaluation of Edited Language Models [15.63231238452797]
Existing editing methods lead to inevitable performance deterioration on general benchmarks.
Instruction-tuned models are more robust to editing, showing less performance drop on general knowledge after editing.
Our findings indicate that current editing methods are only suitable for small-scale knowledge updates within language models.
arXiv Detail & Related papers (2024-10-24T14:36:48Z) - LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models [30.831866499812925]
Large language models (LLMs) require continual knowledge updates to stay abreast of the ever-changing world facts.
We introduce LEMoE, an advanced Mixture of Experts (MoE) adaptor for lifelong model editing.
arXiv Detail & Related papers (2024-06-28T16:17:41Z) - Fundamental Problems With Model Editing: How Should Rational Belief Revision Work in LLMs? [61.68363765350178]
This paper critiques the standard formulation of the model editing problem and proposes a formal testbed for model editing research.
We first describe 12 open problems with model editing, based on challenges with (1) defining the problem, (2) developing benchmarks, and (3) assuming LLMs have editable beliefs in the first place.
Next, we introduce a semi-synthetic dataset for model editing based on Wikidata, where we can evaluate edits against labels given by an idealized Bayesian agent.
arXiv Detail & Related papers (2024-06-27T17:33:03Z) - The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse [58.0132400208411]
Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks.
benchmarking Large Language Models after each edit is impractically time-consuming and resource-intensive.
We have utilized GPT-3.5 to develop a new dataset, HardEdit, based on hard cases.
arXiv Detail & Related papers (2024-02-15T01:50:38Z) - On the Robustness of Editing Large Language Models [57.477943944826904]
Large language models (LLMs) have played a pivotal role in building communicative AI, yet they encounter the challenge of efficient updates.
This work seeks to understand the strengths and limitations of editing methods, facilitating practical applications of communicative AI.
arXiv Detail & Related papers (2024-02-08T17:06:45Z) - Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue [122.20016030723043]
We evaluate the side effects of model editing on large language models (LLMs)
Our analysis reveals that the side effects are caused by model editing altering the original model weights excessively.
To mitigate this, a method named RECT is proposed to regularize the edit update weights.
arXiv Detail & Related papers (2024-01-09T18:03:15Z) - CoditT5: Pretraining for Source Code and Natural Language Editing [34.77621217370665]
CoditT5 is a large language model for software-related editing tasks that is pretrained on large amounts of source code and natural language comments.
We fine-tune it on various downstream editing tasks, including comment updating, bug fixing, and automated code review.
arXiv Detail & Related papers (2022-08-10T16:59:40Z) - Memory-Based Model Editing at Scale [102.28475739907498]
Existing model editors struggle to accurately model an edit's intended scope.
We propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC)
SERAC stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed.
arXiv Detail & Related papers (2022-06-13T23:40:34Z) - Language Anisotropic Cross-Lingual Model Editing [61.51863835749279]
Existing work only studies the monolingual scenario, which lacks the cross-lingual transferability to perform editing simultaneously across languages.
We propose a framework to naturally adapt monolingual model editing approaches to the cross-lingual scenario using parallel corpus.
We empirically demonstrate the failure of monolingual baselines in propagating the edit to multiple languages and the effectiveness of the proposed language anisotropic model editing.
arXiv Detail & Related papers (2022-05-25T11:38:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.