Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs

• URL: http://arxiv.org/abs/2510.22139v1
• Date: Sat, 25 Oct 2025 03:22:59 GMT
• Title: Edit Less, Achieve More: Dynamic Sparse Neuron Masking for Lifelong Knowledge Editing in LLMs
• Authors: Jinzhe Liu, Junshu Sun, Shufan Shen, Chenxue Yang, Shuhui Wang,
• Abstract summary: Lifelong knowledge editing enables continuous, precise updates to outdated knowledge in large language models.<n>Existing methods often accumulate errors throughout the editing process, causing a gradual decline in editing accuracy and generalization.<n>We propose Neuron-Specific Masked Knowledge Editing (NMKE), a novel fine-grained editing framework that combines neuron-level attribution with dynamic sparse masking.
• Score: 36.853517310782344
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Lifelong knowledge editing enables continuous, precise updates to outdated knowledge in large language models (LLMs) without computationally expensive full retraining. However, existing methods often accumulate errors throughout the editing process, causing a gradual decline in both editing accuracy and generalization. To tackle this problem, we propose Neuron-Specific Masked Knowledge Editing (NMKE), a novel fine-grained editing framework that combines neuron-level attribution with dynamic sparse masking. Leveraging neuron functional attribution, we identify two key types of knowledge neurons, with knowledge-general neurons activating consistently across prompts and knowledge-specific neurons activating to specific prompts. NMKE further introduces an entropy-guided dynamic sparse mask, locating relevant neurons to the target knowledge. This strategy enables precise neuron-level knowledge editing with fewer parameter modifications. Experimental results from thousands of sequential edits demonstrate that NMKE outperforms existing methods in maintaining high editing success rates and preserving model general capabilities in lifelong editing.

Related papers

• Massive Editing for Large Language Models Based on Dynamic Weight Generation [51.34392079812964]
  This paper proposes a Massive editing approach for Large Language Models (LLMs) based on dynamic weight Generation (MeG)<n>Our MeG can significantly improve the performance of large-scale knowledge editing in terms of Reliability, Generality, and Locality metrics.
  arXiv  Detail & Related papers  (2025-12-16T13:32:55Z)
• MEMOIR: Lifelong Model Editing with Minimal Overwrite and Informed Retention for LLMs [76.28901550926021]
  Existing methods for lifelong model editing compromise generalization, interfere with past edits, or fail to scale to long editing sequences.<n>We propose MEMOIR, a novel scalable framework that injects knowledge through a residual memory, while preserving the core capabilities of the pre-trained model.<n>MeMOIR achieves state-of-the-art performance across reliability, generalization, and locality metrics, scaling to thousands of sequential edits with minimal forgetting.
  arXiv  Detail & Related papers  (2025-06-09T16:16:42Z)
• CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners [109.87058236007907]
  CaKE (Circuit-aware Knowledge Editing) is a novel method that enhances the effective integration of updated knowledge in large language models.<n> Experiments show that CaKE enables more accurate and consistent use of edited knowledge across related reasoning tasks.
  arXiv  Detail & Related papers  (2025-03-20T17:14:34Z)
• Precise Localization of Memories: A Fine-grained Neuron-level Knowledge Editing Technique for LLMs [47.06544781855325]
  We propose a Fine-grained Neuron-level Knowledge Editing (FiNE) method that enhances editing locality without affecting success rates.<n>By precisely identifying and modifying specific neurons within feed-forward networks, FiNE significantly improves knowledge localization and editing.
  arXiv  Detail & Related papers  (2025-03-03T01:30:28Z)
• GeoEdit: Geometric Knowledge Editing for Large Language Models [52.37408324849593]
  Regular updates are essential for maintaining up-to-date knowledge in large language models (LLMs)<n>We propose a novel framework called Geometric Knowledge Editing (GeoEdit)<n>GeoEdit distinguishes between neurons associated with new knowledge updates and those related to general knowledge perturbations.<n>For the remaining neurons, we integrate both old and new knowledge for aligned directions and apply a "forget-then-learn" editing strategy for opposite directions.
  arXiv  Detail & Related papers  (2025-02-27T10:27:48Z)
• Knowledge Editing for Large Language Model with Knowledge Neuronal Ensemble [13.608354678065222]
  We propose a novel knowledge editing method called Knowledge Neuronal Ensemble (KNE)<n>A knowledge neuronal ensemble represents a group of neurons encoding specific knowledge, thus mitigating the issue of frequent parameter modification.<n> Experimental results on three widely used knowledge editing datasets show that the KNE method significantly improves the accuracy of knowledge editing.
  arXiv  Detail & Related papers  (2024-12-30T00:58:00Z)
• Neuron Empirical Gradient: Discovering and Quantifying Neurons Global Linear Controllability [14.693407823048478]
  We show that the neuron empirical gradient (NEG) captures how changes in activations affect predictions.<n>We also show that NEG effectively captures language skills across diverse prompts through skill neuron probing.<n>Further analysis highlights the key properties of NEG-based skill representation: efficiency, robustness, flexibility, and interdependency.
  arXiv  Detail & Related papers  (2024-12-24T00:01:24Z)
• MEMLA: Enhancing Multilingual Knowledge Editing with Neuron-Masked Low-Rank Adaptation [18.087144677674786]
  We focus on multilingual knowledge editing (MKE), which requires propagating updates across multiple languages. We introduce the Multilingual Knowledge Editing Benchmark (MKEB), a novel dataset comprising 12 languages. We also propose a method that enhances knowledge Editing with neuron-Masked Low-Rank Adaptation (MEMLA)
  arXiv  Detail & Related papers  (2024-06-17T14:03:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.