Balancing Knowledge Updates: Toward Unified Modular Editing in LLMs

• URL: http://arxiv.org/abs/2510.27400v1
• Date: Fri, 31 Oct 2025 11:37:39 GMT
• Title: Balancing Knowledge Updates: Toward Unified Modular Editing in LLMs
• Authors: Jiahao Liu, Zijian Wang, Kuo Zhao, Dong Hu,
• Abstract summary: We propose IntAttn-Edit to update both memory and Attn modules.<n>We show that IntAttn-Edit achieves higher edit success, better generalization, and stronger knowledge preservation than prior methods.
• Score: 24.366567992263303
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge editing has emerged as an efficient approach for updating factual knowledge in large language models (LLMs). It typically locates knowledge storage modules and then modifies their parameters. However, most existing methods focus on the weights of multilayer perceptron (MLP) modules, which are often identified as the main repositories of factual information. Other components, such as attention (Attn) modules, are often ignored during editing. This imbalance can leave residual outdated knowledge and limit editing effectiveness. We perform comprehensive knowledge localization experiments on advanced LLMs and find that Attn modules play a substantial role in factual knowledge storage and retrieval, especially in earlier layers. Based on these insights, we propose IntAttn-Edit, a method that extends the associative memory paradigm to jointly update both MLP and Attn modules. Our approach uses a knowledge balancing strategy that allocates update magnitudes in proportion to each module's measured contribution to knowledge storage. Experiments on standard benchmarks show that IntAttn-Edit achieves higher edit success, better generalization, and stronger knowledge preservation than prior methods. Further analysis shows that the balancing strategy keeps editing performance within an optimal range across diverse settings.

Related papers

• Massive Editing for Large Language Models Based on Dynamic Weight Generation [51.34392079812964]
  This paper proposes a Massive editing approach for Large Language Models (LLMs) based on dynamic weight Generation (MeG)<n>Our MeG can significantly improve the performance of large-scale knowledge editing in terms of Reliability, Generality, and Locality metrics.
  arXiv  Detail & Related papers  (2025-12-16T13:32:55Z)
• Retention analysis of edited knowledge after fine-tuning [5.1877231178075425]
  Large language models (LLMs) store vast amounts of knowledge, which often requires updates to correct factual errors, incorporate newly acquired information, or adapt model behavior.<n>Model editing methods have emerged as efficient solutions for such updates, offering localized and precise knowledge modification at significantly lower computational cost than continual training.<n>However, the effect of fine-tuning on previously edited knowledge remains poorly understood.
  arXiv  Detail & Related papers  (2025-07-14T15:51:19Z)
• Model Merging for Knowledge Editing [53.799891745131724]
  Large Language Models (LLMs) require continuous updates to maintain accurate and current knowledge as the world evolves.<n>Existing knowledge editing approaches offer various solutions for knowledge updating, but they often struggle with sequential editing scenarios.<n>This paper proposes a two-stage framework combining robust supervised fine-tuning (R-SFT) with model merging for knowledge editing.
  arXiv  Detail & Related papers  (2025-06-14T07:42:39Z)
• Disentangling Knowledge Representations for Large Language Model Editing [38.244171146682206]
  We propose DiKE, a novel approach that Disentangles Knowledge representations for LLM Editing.<n>DiKE consists of two key components: a Knowledge Representation Disentanglement (KRD) module that decomposes the subject representation into target-knowledgerelated and -unrelated components, and a Knowledge Edit (DKE) module that updates only the target-related component while explicitly preserving the unrelated one.<n>To rigorously evaluate fine-grained irrelevant knowledge preservation, we construct FINE-KED, a new benchmark comprising fine-grained irrelevant knowledge at different levels of relational similarity to the edited knowledge.
  arXiv  Detail & Related papers  (2025-05-24T16:24:04Z)
• MindBridge: Scalable and Cross-Model Knowledge Editing via Memory-Augmented Modality [55.01380617388064]
  Most existing methods overfit to specific models, causing edited knowledge to be discarded during each update.<n>We introduce MindBridge, a scalable solution inspired by the low coupling between modality processing and LLMs in multi-modal models.<n>MindBridge achieves superior performance even in editing tens of thousands of knowledge entries and can flexibly adapt to different LLMs.
  arXiv  Detail & Related papers  (2025-03-04T15:17:57Z)
• MEMoE: Enhancing Model Editing with Mixture of Experts Adaptors [30.831866499812925]
  MEMoE is a model editing adapter utilizing a Mixture of Experts (MoE) architecture with a knowledge anchor routing strategy. We show the superiority of our approach over both batch editing and sequential batch editing tasks.
  arXiv  Detail & Related papers  (2024-05-29T13:49:44Z)
• Editing Conceptual Knowledge for Large Language Models [65.38231526537476]
  This paper pioneers the investigation of editing conceptual knowledge for Large Language Models (LLMs) We construct a novel benchmark dataset ConceptEdit and establish a suite of new metrics for evaluation. experimental results reveal that, although existing editing methods can efficiently modify concept-level definition to some extent, they also have the potential to distort the related instantial knowledge.
  arXiv  Detail & Related papers  (2024-03-10T16:57:10Z)
• Learning to Edit: Aligning LLMs with Knowledge Editing [101.96620267293731]
  We propose a Learning to Edit (LTE) framework, focusing on teaching large language models to apply updated knowledge into input questions. LTE features a two-phase process: (i) the Alignment Phase, which fine-tunes LLMs on a meticulously curated parallel dataset to make reliable, in-scope edits. We demonstrate LTE's superiority in knowledge editing performance, robustness in both batch and sequential editing, minimal interference on general tasks, and rapid editing speeds.
  arXiv  Detail & Related papers  (2024-02-19T07:45:17Z)
• A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
  Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches. We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
  arXiv  Detail & Related papers  (2024-01-02T16:54:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.