Lifelong Knowledge Editing requires Better Regularization

• URL: http://arxiv.org/abs/2502.01636v2
• Date: Wed, 21 May 2025 17:58:23 GMT
• Title: Lifelong Knowledge Editing requires Better Regularization
• Authors: Akshat Gupta, Phudish Prateepamornkul, Maochuan Lu, Ahmed Alaa, Thomas Hartvigsen, Gopala Anumanchipalli,
• Abstract summary: We formalize the popular locate-then-edit methods as a two-step fine-tuning process.<n>We show that model degradation occurs due to over-optimization of internal activations and continuous norm-growth of edited matrices.<n>Applying these simple yet effective regularization techniques at key points in the editing process can substantially mitigate model degradation.
• Score: 11.14177136208272
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge editing is a promising way to improve factuality in large language models, but recent studies have shown significant model degradation during sequential editing. In this paper, we formalize the popular locate-then-edit methods as a two-step fine-tuning process, allowing us to precisely identify the root cause of this degradation. We show that model degradation occurs due to (1) over-optimization of internal activations and (2) continuous norm-growth of edited matrices. To mitigate these issues, we introduce two regularization techniques: (1) Most-Probable Early Stopping (MPES) and (2) explicit Frobenius norm-constraint. We demonstrate that applying these simple yet effective regularization techniques at key points in the editing process can substantially mitigate model degradation. Combining these regularization methods enables scaling locate-then-edit methods to 10,000 edits while reducing editing time by 42-61%. These results show that targeted regularization is essential for lifelong knowledge editing.

Related papers

• How Robust is Model Editing after Fine-Tuning? An Empirical Study on Text-to-Image Diffusion Models [7.342540592387184]
  We investigate the interaction between model editing and fine-tuning in the context of T2I diffusion models.<n>Our findings reveal a trend: edits generally fail to persist through fine-tuning, even when fine-tuning is tangential or unrelated to the edits.<n>These findings highlight the need for more robust techniques to ensure reliable long-term control and alignment of deployed AI systems.
  arXiv  Detail & Related papers  (2025-06-23T09:10:29Z)
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs [82.34547399693966]
  Existing methods for lifelong model editing compromise generalization, interfere with past edits, or fail to scale to long editing sequences.<n>We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory.<n>MeMOIR confines each edit to a distinct subset of the memory parameters, minimizing interference among edits.
  arXiv  Detail & Related papers  (2025-06-09T16:16:42Z)
• LyapLock: Bounded Knowledge Preservation in Sequential Large Language Model Editing [27.918524905286475]
  Current locate-then-edit approaches exhibit a progressive performance decline during sequential editing.<n>textbfLyapLock is proposed to decompose the long-term constrained programming into tractable stepwise subproblems for efficient solving.<n> Experimental results show that our framework scales sequential editing capacity to over 10,000 edits while stabilizing general capabilities and boosting average editing efficacy by 11.89% over SOTA baselines.
  arXiv  Detail & Related papers  (2025-05-21T16:16:33Z)
• DeltaEdit: Enhancing Sequential Editing in Large Language Models by Controlling Superimposed Noise [1.2697731449512988]
  Sequential knowledge editing techniques aim to continuously update the knowledge in large language models at a low cost.<n>Existing sequential editing methods suffer from a significant decline in editing success rates after long-term editing.<n>We propose DeltaEdit, a novel method that reduces interference between edits to mitigate deviation.<n> Experimental results demonstrate that DeltaEdit significantly outperforms existing methods in edit success rates and the retention of generalization capabilities.
  arXiv  Detail & Related papers  (2025-05-12T07:11:26Z)
• Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model Editing [7.752740499342269]
  Large Language Models (LLMs) are widely deployed in downstream tasks, but keeping their knowledge up-to-date via retraining or fine-tuning is often computationally expensive.<n>Model editing provides a more efficient alternative by updating a targeted subset of parameters, which often follows the locate-and-edit paradigm.<n>We propose two complementary methods: iterative model editing, which applies successive edits to mitigate UnderEdit, and neighbor-assisted model editing, which incorporates neighboring knowledge during editing to reduce OverEdit.
  arXiv  Detail & Related papers  (2025-03-14T21:53:12Z)
• Understanding the Limits of Lifelong Knowledge Editing in LLMs [59.12302872055081]
  We bridge research into lifelong knowledge editing to real-world edits at practically relevant scale. We first introduce WikiBigEdit; a large-scale benchmark of real-world Wikidata edits. In its first instance, it includes over 500K question-answer pairs for knowledge editing.
  arXiv  Detail & Related papers  (2025-03-07T18:45:42Z)
• Constraining Sequential Model Editing with Editing Anchor Compression [40.93064933191375]
  Large language models (LLMs) struggle with hallucinations due to false or outdated knowledge. This paper statistically observes that the parameter matrix after editing exhibits a significant deviation compared to its previous state as the number of edits increases. A framework termed Editing Anchor Compression (EAC) is proposed to constrain the deviation of the parameter matrix during sequential editing.
  arXiv  Detail & Related papers  (2025-02-25T03:56:49Z)
• The Mirage of Model Editing: Revisiting Evaluation in the Wild [70.17413507444704]
  We introduce QAEdit, a new benchmark aligned with widely used question answering (QA) datasets, and WILD, a task-agnostic evaluation framework.<n>Our single editing experiments show that current editing methods perform substantially worse than previously reported.
  arXiv  Detail & Related papers  (2025-02-16T15:57:55Z)
• AnyEdit: Edit Any Knowledge Encoded in Language Models [69.30638272162267]
  We propose AnyEdit, a new autoregressive editing paradigm for large language models (LLMs) It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs. It outperforms strong baselines by 21.5% on benchmarks including UnKEBench, AKEW, and our new EditEverything dataset for long-form diverse-formatted knowledge.
  arXiv  Detail & Related papers  (2025-02-08T16:18:37Z)
• O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing [0.0]
  Large language models (LLMs) acquire knowledge during pre-training, but over time, this knowledge may become incorrect or outdated, necessitating updates after training. We propose Orthogonal Subspace Editing, O-Edit. This algorithmizes the direction of each knowledge update, minimizing interference between successive updates and reducing the impact of new updates on unrelated knowledge. It can perform thousands of edits on mainstream LLMs, achieving an average performance improvement that is 4.2 times better than existing methods while effectively preserving the model's performance on downstream tasks, all with minimal additional parameter overhead.
  arXiv  Detail & Related papers  (2024-10-15T10:16:45Z)
• Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization [48.07144492109635]
  Large language models need to be updated regularly. Model editing is challenging as it might also affect knowledge that is unrelated to the new data. We propose SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization.
  arXiv  Detail & Related papers  (2024-10-03T12:28:13Z)
• AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models [65.93240009586351]
  Large language models (LLMs) often exhibit hallucinations due to incorrect or outdated knowledge. We introduce AlphaEdit, a novel solution that projects perturbation onto the null space of the preserved knowledge before applying it to the parameters. We theoretically prove that this projection ensures the output of post-edited LLMs remains unchanged when queried about the preserved knowledge.
  arXiv  Detail & Related papers  (2024-10-03T10:06:27Z)
• Zero-Shot Video Editing through Adaptive Sliding Score Distillation [51.57440923362033]
  This study proposes a novel paradigm of video-based score distillation, facilitating direct manipulation of original video content. We propose an Adaptive Sliding Score Distillation strategy, which incorporates both global and local video guidance to reduce the impact of editing errors.
  arXiv  Detail & Related papers  (2024-06-07T12:33:59Z)
• Perturbation-Restrained Sequential Model Editing [33.51709226068619]
  Current model editing methods compromise the general abilities of large language models (LLMs) as the number of edits increases. A framework termed Perturbation Restraint on Upper bouNd for Editing (PRUNE) is proposed, which applies the condition number restraints in sequential editing. The results show that PRUNE can preserve general abilities while maintaining the editing performance effectively in sequential model editing.
  arXiv  Detail & Related papers  (2024-05-27T04:40:56Z)
• WISE: Rethinking the Knowledge Memory for Lifelong Model Editing of Large Language Models [78.22291694903659]
  Large language models (LLMs) need knowledge updates to meet the ever-growing world facts and correct the hallucinated responses. Where the updated knowledge resides in memories is a fundamental question for model editing. We propose WISE to bridge the gap between memories.
  arXiv  Detail & Related papers  (2024-05-23T16:35:52Z)
• Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing [2.569159339315845]
  We show that disabling edits are an artifact of irregularities in the implementation of Rank-One Model Editing (ROME) We provide a more stable implementation ROME, which we call r-ROME, and show that model collapse is no longer observed when making large scale sequential edits with r-ROME.
  arXiv  Detail & Related papers  (2024-03-11T21:33:05Z)
• WilKE: Wise-Layer Knowledge Editor for Lifelong Knowledge Editing [19.357663224043534]
  This study reveals a performance degradation encountered by knowledge editing in lifelong editing. We introduce a knowledge editing approach named Wise-Layer Knowledge Editor (WilKE) WilKE selects editing layer based on the pattern matching degree of editing knowledge across different layers in language models.
  arXiv  Detail & Related papers  (2024-02-16T05:29:59Z)
• The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse [58.0132400208411]
  Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. benchmarking Large Language Models after each edit is impractically time-consuming and resource-intensive. We have utilized GPT-3.5 to develop a new dataset, HardEdit, based on hard cases.
  arXiv  Detail & Related papers  (2024-02-15T01:50:38Z)
• DeepEdit: Knowledge Editing as Decoding with Constraints [118.78008395850888]
  How to edit the knowledge in multi-step reasoning has become the major challenge in the knowledge editing (KE) of large language models (LLMs) We propose a new KE framework: DEEPEDIT, which enhances LLMs's ability to generate coherent reasoning chains with new knowledge through depth-first search. In addition to DEEPEDIT, we propose two new KE benchmarks: MQUAKE-2002 and MQUAKE-HARD, which provide more precise and challenging assessments of KE approaches.
  arXiv  Detail & Related papers  (2024-01-19T03:48:27Z)
• Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue [122.20016030723043]
  We evaluate the side effects of model editing on large language models (LLMs) Our analysis reveals that the side effects are caused by model editing altering the original model weights excessively. To mitigate this, a method named RECT is proposed to regularize the edit update weights.
  arXiv  Detail & Related papers  (2024-01-09T18:03:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.