Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model Editing

• URL: http://arxiv.org/abs/2503.11895v2
• Date: Tue, 17 Jun 2025 19:33:26 GMT
• Title: Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model Editing
• Authors: Bhiman Kumar Baghel, Scott M. Jordan, Zheyuan Ryan Shi, Xiang Lorraine Li,
• Abstract summary: Large Language Models (LLMs) are widely deployed in downstream tasks, but keeping their knowledge up-to-date via retraining or fine-tuning is often computationally expensive.<n>Model editing provides a more efficient alternative by updating a targeted subset of parameters, which often follows the locate-and-edit paradigm.<n>We propose two complementary methods: iterative model editing, which applies successive edits to mitigate UnderEdit, and neighbor-assisted model editing, which incorporates neighboring knowledge during editing to reduce OverEdit.
• Score: 7.752740499342269
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models (LLMs) are widely deployed in downstream tasks, but keeping their knowledge up-to-date via retraining or fine-tuning is often computationally expensive. Model editing provides a more efficient alternative by updating a targeted subset of parameters, which often follows the locate-and-edit paradigm. Despite this efficiency, existing methods are limited: edits may fail to inject knowledge (UnderEdit) or unintentionally disrupt unrelated neighboring knowledge (OverEdit). To address these challenges, we propose two complementary methods: iterative model editing, which applies successive edits to mitigate UnderEdit, and neighbor-assisted model editing, which incorporates neighboring knowledge during editing to reduce OverEdit. Our extensive experiments show that these techniques improve editing performance across multiple LLMs, algorithms, and benchmarks, reducing UnderEdit by up to 38 percentage points and OverEdit by up to 6, while remaining broadly applicable to any locate-and-edit method.

Related papers

• InComeS: Integrating Compression and Selection Mechanisms into LLMs for Efficient Model Editing [77.47790551485721]
  In-context learning is a promising editing method by comprehending edit information through context encoding.<n>This method is constrained by the limited context window of large language models.<n>We propose InComeS, a flexible framework that enhances LLMs' ability to process editing contexts.
  arXiv  Detail & Related papers  (2025-05-28T09:20:18Z)
• DeltaEdit: Enhancing Sequential Editing in Large Language Models by Controlling Superimposed Noise [1.2697731449512988]
  Sequential knowledge editing techniques aim to continuously update the knowledge in large language models at a low cost.<n>Existing sequential editing methods suffer from a significant decline in editing success rates after long-term editing.<n>We propose DeltaEdit, a novel method that reduces interference between edits to mitigate deviation.<n> Experimental results demonstrate that DeltaEdit significantly outperforms existing methods in edit success rates and the retention of generalization capabilities.
  arXiv  Detail & Related papers  (2025-05-12T07:11:26Z)
• AnyEdit: Edit Any Knowledge Encoded in Language Models [69.30638272162267]
  We propose AnyEdit, a new autoregressive editing paradigm for large language models (LLMs)<n>It decomposes long-form knowledge into sequential chunks and iteratively edits the key token in each chunk, ensuring consistent and accurate outputs.<n>It outperforms strong baselines by 21.5% on benchmarks including UnKEBench, AKEW, and our new EditEverything dataset for long-form diverse-formatted knowledge.
  arXiv  Detail & Related papers  (2025-02-08T16:18:37Z)
• O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing [0.0]
  Large language models (LLMs) acquire knowledge during pre-training, but over time, this knowledge may become incorrect or outdated, necessitating updates after training. We propose Orthogonal Subspace Editing, O-Edit. This algorithmizes the direction of each knowledge update, minimizing interference between successive updates and reducing the impact of new updates on unrelated knowledge. It can perform thousands of edits on mainstream LLMs, achieving an average performance improvement that is 4.2 times better than existing methods while effectively preserving the model's performance on downstream tasks, all with minimal additional parameter overhead.
  arXiv  Detail & Related papers  (2024-10-15T10:16:45Z)
• Uncovering Overfitting in Large Language Model Editing [35.55260822503773]
  We identify and investigate the phenomenon of Editing Overfit, where edited models assign disproportionately high probabilities to the edit target.<n>We propose Learn the Inference (LTI), which introduce a Multi-stage Inference Constraint module to guide the edited models in recalling new knowledge.
  arXiv  Detail & Related papers  (2024-10-10T11:09:00Z)
• Neuron-Level Sequential Editing for Large Language Models [19.324852774144752]
  We introduce textbfNeuron-level textbfSequential textbfEditing (NSE) for supporting sequential model editing. Specifically, we optimize the target layer's hidden states using the model's original weights to prevent model failure. Our experiments demonstrate that NSE significantly outperforms current modifying parameters model editing methods.
  arXiv  Detail & Related papers  (2024-10-05T05:52:22Z)
• Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization [48.07144492109635]
  Large language models need to be updated regularly. Model editing is challenging as it might also affect knowledge that is unrelated to the new data. We propose SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization.
  arXiv  Detail & Related papers  (2024-10-03T12:28:13Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors. Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update. We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• Is Bigger Edit Batch Size Always Better? -- An Empirical Study on Model Editing with Llama-3 [2.569159339315845]
  This study presents a targeted model editing analysis focused on the latest large language model, Llama-3. We identify the most effective layers for targeted edits through an evaluation that encompasses up to 4096 edits.
  arXiv  Detail & Related papers  (2024-05-01T17:50:37Z)
• Rebuilding ROME : Resolving Model Collapse during Sequential Model Editing [2.569159339315845]
  We show that disabling edits are an artifact of irregularities in the implementation of Rank-One Model Editing (ROME) We provide a more stable implementation ROME, which we call r-ROME, and show that model collapse is no longer observed when making large scale sequential edits with r-ROME.
  arXiv  Detail & Related papers  (2024-03-11T21:33:05Z)
• The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse [58.0132400208411]
  Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. benchmarking Large Language Models after each edit is impractically time-consuming and resource-intensive. We have utilized GPT-3.5 to develop a new dataset, HardEdit, based on hard cases.
  arXiv  Detail & Related papers  (2024-02-15T01:50:38Z)
• Model Editing Harms General Abilities of Large Language Models: Regularization to the Rescue [122.20016030723043]
  We evaluate the side effects of model editing on large language models (LLMs) Our analysis reveals that the side effects are caused by model editing altering the original model weights excessively. To mitigate this, a method named RECT is proposed to regularize the edit update weights.
  arXiv  Detail & Related papers  (2024-01-09T18:03:15Z)
• Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors [53.819805242367345]
  We propose GRACE, a lifelong model editing method, which implements spot-fixes on streaming errors of a deployed model. GRACE writes new mappings into a pre-trained model's latent space, creating a discrete, local codebook of edits without altering model weights. Our experiments on T5, BERT, and GPT models show GRACE's state-of-the-art performance in making and retaining edits, while generalizing to unseen inputs.
  arXiv  Detail & Related papers  (2022-11-20T17:18:22Z)
• Memory-Based Model Editing at Scale [102.28475739907498]
  Existing model editors struggle to accurately model an edit's intended scope. We propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC) SERAC stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed.
  arXiv  Detail & Related papers  (2022-06-13T23:40:34Z)
• Learning Structural Edits via Incremental Tree Transformations [102.64394890816178]
  We present a generic model for incremental editing of structured data (i.e., "structural edits") Our editor learns to iteratively generate tree edits (e.g., deleting or adding a subtree) and applies them to the partially edited data. We evaluate our proposed editor on two source code edit datasets, where results show that, with the proposed edit encoder, our editor significantly improves accuracy over previous approaches.
  arXiv  Detail & Related papers  (2021-01-28T16:11:32Z)
• A Structural Model for Contextual Code Changes [20.185486717922615]
  Given a code snippet that is partially edited, our goal is to predict a completion of the edit for the rest of the snippet. Our model achieves a 28% relative gain over state-of-the-art sequential models and 2x higher accuracy than syntactic models that learn to generate the edited code.
  arXiv  Detail & Related papers  (2020-05-27T07:16:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.