One for All: Update Parameterized Knowledge Across Multiple Models

• URL: http://arxiv.org/abs/2506.00817v1
• Date: Sun, 01 Jun 2025 03:48:54 GMT
• Title: One for All: Update Parameterized Knowledge Across Multiple Models
• Authors: Weitao Ma, Xiyuan Du, Xiaocheng Feng, Lei Huang, Yichong Huang, Huiyi Zhang, Xiaoliang Yang, Baohang Li, Xiachong Feng, Ting Liu, Bing Qin,
• Abstract summary: Large language models (LLMs) encode vast world knowledge but struggle to stay up-to-date, often leading to errors and hallucinations.<n> Knowledge editing offers an efficient alternative to retraining, enabling targeted modifications by updating specific model parameters.<n>We propose OnceEdit, a novel ensemble-based approach that employs a plug-in model as the editing module.
• Score: 35.137065486616805
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) encode vast world knowledge but struggle to stay up-to-date, often leading to errors and hallucinations. Knowledge editing offers an efficient alternative to retraining, enabling targeted modifications by updating specific model parameters. However, existing methods primarily focus on individual models, posing challenges in efficiently updating multiple models and adapting to new models. To address this, we propose OnceEdit, a novel ensemble-based approach that employs a plug-in model as the editing module, enabling stable knowledge updates across multiple models. Building on the model ensemble, OnceEdit introduces two key mechanisms to enhance its effectiveness. First, we introduce a dynamic weight mechanism through a \weight token for distinguishing between edit-related and non-edit-related instances, ensuring the appropriate utilization of knowledge from integrated models. Second, we incorporate an ensemble enhancement mechanism to mitigate the excessive reliance on the central model inherent in the model ensemble technique, making it more suitable for knowledge editing. Extensive experiments on diverse LLMs demonstrate that OnceEdit consistently outperforms existing methods while achieving superior editing efficiency. Further analysis confirms its adaptability and stability in multi-model editing scenarios. Our code will be available.

Related papers

• DualEdit: Dual Editing for Knowledge Updating in Vision-Language Models [26.762431651154607]
  We propose DualEdit, an editor that modifies both textual and visual modalities at their respective key layers.<n>We evaluate DualEdit across multiple VLM backbones and benchmark datasets, demonstrating its superiority over state-of-the-art VLM editing baselines.
  arXiv  Detail & Related papers  (2025-06-16T16:04:16Z)
• Model Merging for Knowledge Editing [53.799891745131724]
  Large Language Models (LLMs) require continuous updates to maintain accurate and current knowledge as the world evolves.<n>Existing knowledge editing approaches offer various solutions for knowledge updating, but they often struggle with sequential editing scenarios.<n>This paper proposes a two-stage framework combining robust supervised fine-tuning (R-SFT) with model merging for knowledge editing.
  arXiv  Detail & Related papers  (2025-06-14T07:42:39Z)
• BalancEdit: Dynamically Balancing the Generality-Locality Trade-off in Multi-modal Model Editing [18.40863022476747]
  We introduce the concept of the generality-locality trade-off in multi-modal model editing.<n>We propose BalancEdit, a novel method for balanced model editing.<n>Our results confirm the effectiveness of BalancEdit, demonstrating minimal trade-offs while maintaining robust editing capabilities.
  arXiv  Detail & Related papers  (2025-05-02T15:31:32Z)
• Resolving UnderEdit & OverEdit with Iterative & Neighbor-Assisted Model Editing [7.752740499342269]
  Large Language Models (LLMs) are used in various downstream language tasks.<n>Both retraining and fine-tuning the model can be costly.<n>Model editing offers an efficient and effective alternative by a single update to only a key subset of model parameters.<n>We propose iterative model editing, based on our hypothesis that a single parameter update is often insufficient.<n>Our methods effectively reduce UnderEdit up to 38 percentage points and OverEdit up to 6 percentage points across multiple model editing algorithms, LLMs, and benchmark datasets.
  arXiv  Detail & Related papers  (2025-03-14T21:53:12Z)
• Neuron-Level Sequential Editing for Large Language Models [19.324852774144752]
  We introduce textbfNeuron-level textbfSequential textbfEditing (NSE) for supporting sequential model editing. Specifically, we optimize the target layer's hidden states using the model's original weights to prevent model failure. Our experiments demonstrate that NSE significantly outperforms current modifying parameters model editing methods.
  arXiv  Detail & Related papers  (2024-10-05T05:52:22Z)
• Better Call SAUL: Fluent and Consistent Language Model Editing with Generation Regularization [48.07144492109635]
  Large language models need to be updated regularly. Model editing is challenging as it might also affect knowledge that is unrelated to the new data. We propose SAUL, a streamlined model editing method that uses sentence concatenation with augmented random facts for generation regularization.
  arXiv  Detail & Related papers  (2024-10-03T12:28:13Z)
• ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA [55.697627106315004]
  Large language models (LLMs) require model editing to efficiently update specific knowledge within them and avoid factual errors.<n>Previous approaches manage sequential edits by freezing original parameters and discretely allocating new parameters for each knowledge update.<n>We propose ELDER, a novel approach to create a continuous association between data and adapters.
  arXiv  Detail & Related papers  (2024-08-19T02:27:00Z)
• Consecutive Batch Model Editing with HooK Layers [59.673084839708224]
  CoachHooK is a model editing method that simultaneously supports sequential and batch editing. It is memory-friendly as it only needs a small amount of it to store several hook layers whose size remains unchanged over time.
  arXiv  Detail & Related papers  (2024-03-08T14:07:44Z)
• The Butterfly Effect of Model Editing: Few Edits Can Trigger Large Language Models Collapse [58.0132400208411]
  Even a single edit can trigger model collapse, manifesting as significant performance degradation in various benchmark tasks. benchmarking Large Language Models after each edit is impractically time-consuming and resource-intensive. We have utilized GPT-3.5 to develop a new dataset, HardEdit, based on hard cases.
  arXiv  Detail & Related papers  (2024-02-15T01:50:38Z)
• Memory-Based Model Editing at Scale [102.28475739907498]
  Existing model editors struggle to accurately model an edit's intended scope. We propose Semi-Parametric Editing with a Retrieval-Augmented Counterfactual Model (SERAC) SERAC stores edits in an explicit memory and learns to reason over them to modulate the base model's predictions as needed.
  arXiv  Detail & Related papers  (2022-06-13T23:40:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.