DeltaEdit: Enhancing Sequential Editing in Large Language Models by Controlling Superimposed Noise

• URL: http://arxiv.org/abs/2505.07899v1
• Date: Mon, 12 May 2025 07:11:26 GMT
• Title: DeltaEdit: Enhancing Sequential Editing in Large Language Models by Controlling Superimposed Noise
• Authors: Ding Cao, Yuchen Cai, Rongxi Guo, Xuesong He, Guiquan Liu,
• Abstract summary: Sequential knowledge editing techniques aim to continuously update the knowledge in large language models at a low cost.<n>Existing sequential editing methods suffer from a significant decline in editing success rates after long-term editing.<n>We propose DeltaEdit, a novel method that reduces interference between edits to mitigate deviation.<n> Experimental results demonstrate that DeltaEdit significantly outperforms existing methods in edit success rates and the retention of generalization capabilities.
• Score: 1.2697731449512988
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Sequential knowledge editing techniques aim to continuously update the knowledge in large language models at a low cost, preventing the models from generating outdated or incorrect information. However, existing sequential editing methods suffer from a significant decline in editing success rates after long-term editing. Through theoretical analysis and experiments, we identify that as the number of edits increases, the model's output increasingly deviates from the desired target, leading to a drop in editing success rates. We refer to this issue as the accumulation of superimposed noise problem. To address this, we identify the factors contributing to this deviation and propose DeltaEdit, a novel method that optimizes update parameters through a dynamic orthogonal constraints strategy, effectively reducing interference between edits to mitigate deviation. Experimental results demonstrate that DeltaEdit significantly outperforms existing methods in edit success rates and the retention of generalization capabilities, ensuring stable and reliable model performance even under extensive sequential editing.

Related papers

• InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing [77.47790551485721]
  In-context learning is a promising editing method by comprehending edit information through context encoding.<n>This method is constrained by the limited context window of large language models.<n>We propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts.
  arXiv  Detail & Related papers  (2025-05-28T09:20:18Z)
• LyapLock: Bounded Knowledge Preservation in Sequential Large Language Model Editing [27.918524905286475]
  Current locate-then-edit approaches exhibit a progressive performance decline during sequential editing.<n>textbfLyapLock is proposed to decompose the long-term constrained programming into tractable stepwise subproblems for efficient solving.<n> Experimental results show that our framework scales sequential editing capacity to over 10,000 edits while stabilizing general capabilities and boosting average editing efficacy by 11.89% over SOTA baselines.
  arXiv  Detail & Related papers  (2025-05-21T16:16:33Z)
• Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model Editing [7.752740499342269]
  Large Language Models (LLMs) are used in various downstream language tasks.<n>Both retraining and fine-tuning the model can be costly.<n>Model editing offers an efficient and effective alternative by a single update to only a key subset of model parameters.<n>We propose iterative model editing, based on our hypothesis that a single parameter update is often insufficient.<n>Our methods effectively reduce UnderEdit up to 38 percentage points and OverEdit up to 6 percentage points across multiple model editing algorithms, LLMs, and benchmark datasets.
  arXiv  Detail & Related papers  (2025-03-14T21:53:12Z)
• The Mirage of Model Editing: Revisiting Evaluation in the Wild [70.17413507444704]
  We introduce QAEdit, a new benchmark aligned with widely used question answering (QA) datasets, and WILD, a task-agnostic evaluation framework.<n>Our single editing experiments show that current editing methods perform substantially worse than previously reported.
  arXiv  Detail & Related papers  (2025-02-16T15:57:55Z)
• Reasons and Solutions for the Decline in Model Performance after Editing [17.756172082400163]
  This paper explores the reasons for the performance decline of the edited model and optimize the editing method. The performance of the editing model is mainly affected by the diversity of editing targets and sequence length. In order to improve the performance of the editing model, this paper proposes a Dump for Sequence (D4S) method.
  arXiv  Detail & Related papers  (2024-10-31T11:49:44Z)
• Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization [48.07144492109635]
  Large language models need to be updated regularly. Model editing is challenging as it might also affect knowledge that is unrelated to the new data. We propose SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization.
  arXiv  Detail & Related papers  (2024-10-03T12:28:13Z)
• Efficiently Quantifying and Mitigating Ripple Effects in Model Editing [27.627105709896025]
  Large Language Models are crucial for rectifying outdated or erroneous information. editing these models often leads to a complex issue known as the ripple effect in the hidden space. This paper proposes a novel evaluation methodology, which quantitatively evaluates the adaptations of the model and the subsequent impact of editing. Furthermore, we introduce the Selective Impact Revision(SIR), a model editing method designed to mitigate this ripple effect.
  arXiv  Detail & Related papers  (2024-03-12T17:04:28Z)
• The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse [58.0132400208411]
  Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. benchmarking Large Language Models after each edit is impractically time-consuming and resource-intensive. We have utilized GPT-3.5 to develop a new dataset, HardEdit, based on hard cases.
  arXiv  Detail & Related papers  (2024-02-15T01:50:38Z)
• Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue [122.20016030723043]
  We evaluate the side effects of model editing on large language models (LLMs) Our analysis reveals that the side effects are caused by model editing altering the original model weights excessively. To mitigate this, a method named RECT is proposed to regularize the edit update weights.
  arXiv  Detail & Related papers  (2024-01-09T18:03:15Z)
• Edit at your own risk: evaluating the robustness of edited models to distribution shifts [0.0]
  We investigate how model editing affects the general robustness of a model, as well as the robustness of the specific behavior targeted by the edit. We find that edits tend to reduce general robustness, but that the degree of degradation depends on the editing algorithm and layers chosen. Motivated by these observations we introduce a new model editing algorithm, 1-layer (1-LI), which uses weight-space to navigate the trade-off between editing task accuracy and general robustness.
  arXiv  Detail & Related papers  (2023-02-28T19:41:37Z)
• Memory-Based Model Editing at Scale [102.28475739907498]
  Existing model editors struggle to accurately model an edit's intended scope. We propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC) SERAC stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed.
  arXiv  Detail & Related papers  (2022-06-13T23:40:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.