GeoEdit: Geometric Knowledge Editing for Large Language Models

• URL: http://arxiv.org/abs/2502.19953v1
• Date: Thu, 27 Feb 2025 10:27:48 GMT
• Title: GeoEdit: Geometric Knowledge Editing for Large Language Models
• Authors: Yujie Feng, Liming Zhan, Zexin Lu, Yongxin Xu, Xu Chu, Yasha Wang, Jiannong Cao, Philip S. Yu, Xiao-Ming Wu,
• Abstract summary: Regular updates are essential for maintaining up-to-date knowledge in large language models (LLMs)<n>We propose a novel framework called Geometric Knowledge Editing (GeoEdit)<n>GeoEdit distinguishes between neurons associated with new knowledge updates and those related to general knowledge perturbations.<n>For the remaining neurons, we integrate both old and new knowledge for aligned directions and apply a "forget-then-learn" editing strategy for opposite directions.
• Score: 52.37408324849593
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Regular updates are essential for maintaining up-to-date knowledge in large language models (LLMs). Consequently, various model editing methods have been developed to update specific knowledge within LLMs. However, training-based approaches often struggle to effectively incorporate new knowledge while preserving unrelated general knowledge. To address this challenge, we propose a novel framework called Geometric Knowledge Editing (GeoEdit). GeoEdit utilizes the geometric relationships of parameter updates from fine-tuning to differentiate between neurons associated with new knowledge updates and those related to general knowledge perturbations. By employing a direction-aware knowledge identification method, we avoid updating neurons with directions approximately orthogonal to existing knowledge, thus preserving the model's generalization ability. For the remaining neurons, we integrate both old and new knowledge for aligned directions and apply a "forget-then-learn" editing strategy for opposite directions. Additionally, we introduce an importance-guided task vector fusion technique that filters out redundant information and provides adaptive neuron-level weighting, further enhancing model editing performance. Extensive experiments on two publicly available datasets demonstrate the superiority of GeoEdit over existing state-of-the-art methods.

Related papers

• Knowledge Graph Enhanced Generative Multi-modal Models for Class-Incremental Learning [51.0864247376786]
  We introduce a Knowledge Graph Enhanced Generative Multi-modal model (KG-GMM) that builds an evolving knowledge graph throughout the learning process. During testing, we propose a Knowledge Graph Augmented Inference method that locates specific categories by analyzing relationships within the generated text.
  arXiv  Detail & Related papers  (2025-03-24T07:20:43Z)
• CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners [88.35958039968081]
  CaKE (Circuit-aware Knowledge Editing) is a novel method that enables more effective integration of updated knowledge in large language models. Results show that CaKE enables more accurate and consistent use of updated knowledge across related reasoning tasks.
  arXiv  Detail & Related papers  (2025-03-20T17:14:34Z)
• Knowledge Editing for Large Language Model with Knowledge Neuronal Ensemble [13.608354678065222]
  We propose a novel knowledge editing method called Knowledge Neuronal Ensemble (KNE) A knowledge neuronal ensemble represents a group of neurons encoding specific knowledge, thus mitigating the issue of frequent parameter modification. Experimental results on three widely used knowledge editing datasets show that the KNE method significantly improves the accuracy of knowledge editing.
  arXiv  Detail & Related papers  (2024-12-30T00:58:00Z)
• Stable Knowledge Editing in Large Language Models [68.98582618305679]
  We introduce StableKE, a knowledge editing method based on knowledge augmentation rather than knowledge localization. To overcome the expense of human labeling, StableKE integrates two automated knowledge augmentation strategies. StableKE surpasses other knowledge editing methods, demonstrating stability both edited knowledge and multi-hop knowledge.
  arXiv  Detail & Related papers  (2024-02-20T14:36:23Z)
• A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
  Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches. We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
  arXiv  Detail & Related papers  (2024-01-02T16:54:58Z)
• History Matters: Temporal Knowledge Editing in Large Language Model [42.74144542674756]
  We introduce the task of Temporal Knowledge Editing (TKE) and establish a benchmark AToKe to evaluate current model editing methods. We find that while existing model editing methods are effective at making models remember new knowledge, the edited model catastrophically forgets historical knowledge. To address this gap, we propose a simple and general framework termed Multi-Editing with Time Objective (METO) for enhancing existing editing models.
  arXiv  Detail & Related papers  (2023-12-09T07:51:56Z)
• Continual Learning for Natural Language Generation in Task-oriented Dialog Systems [72.92029584113676]
  Natural language generation (NLG) is an essential component of task-oriented dialog systems. We study NLG in a "continual learning" setting to expand its knowledge to new domains or functionalities incrementally. The major challenge towards this goal is catastrophic forgetting, meaning that a continually trained model tends to forget the knowledge it has learned before.
  arXiv  Detail & Related papers  (2020-10-02T10:32:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.