CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners

• URL: http://arxiv.org/abs/2503.16356v2
• Date: Tue, 23 Sep 2025 17:10:14 GMT
• Title: CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners
• Authors: Yunzhi Yao, Jizhan Fang, Jia-Chen Gu, Ningyu Zhang, Shumin Deng, Huajun Chen, Nanyun Peng,
• Abstract summary: CaKE (Circuit-aware Knowledge Editing) is a novel method that enhances the effective integration of updated knowledge in large language models.<n> Experiments show that CaKE enables more accurate and consistent use of edited knowledge across related reasoning tasks.
• Score: 109.87058236007907
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Knowledge Editing (KE) enables the modification of outdated or incorrect information in large language models (LLMs). While existing KE methods can update isolated facts, they often fail to generalize these updates to multi-hop reasoning tasks that rely on the modified knowledge. Through an analysis of reasoning circuits -- the neural pathways LLMs use for knowledge-based inference, we find that current layer-localized KE approaches (e.g., MEMIT, WISE), which edit only single or a few model layers, inadequately integrate updated knowledge into these reasoning pathways. To address this limitation, we present CaKE (Circuit-aware Knowledge Editing), a novel method that enhances the effective integration of updated knowledge in LLMs. By only leveraging a few curated data samples guided by our circuit-based analysis, CaKE stimulates the model to develop appropriate reasoning circuits for newly incorporated knowledge. Experiments show that CaKE enables more accurate and consistent use of edited knowledge across related reasoning tasks, achieving an average improvement of 20% in multi-hop reasoning accuracy on the MQuAKE dataset while requiring less memory than existing KE methods. We release the code and data in https://github.com/zjunlp/CaKE.

Related papers

• Retention analysis of edited knowledge after fine-tuning [5.440397659472036]
  Large language models (LLMs) store vast amounts of knowledge, which often requires updates to correct factual errors, incorporate newly acquired information, or adapt model behavior.<n>Model editing methods have emerged as efficient solutions for such updates, offering localized and precise knowledge modification at significantly lower computational cost than continual training.<n>However, the effect of fine-tuning on previously edited knowledge remains poorly understood.
  arXiv  Detail & Related papers  (2025-07-14T15:51:19Z)
• Decoupling Reasoning and Knowledge Injection for In-Context Knowledge Editing [12.5122702720856]
  In-context editing (ICE) offers a lightweight solution by injecting new knowledge directly into the input context.<n>Existing ICE approaches do not explicitly separate the newly injected knowledge from the model's original reasoning process.<n>We propose DecKER, a novel ICE framework that decouples reasoning from knowledge editing by generating a masked reasoning path.
  arXiv  Detail & Related papers  (2025-05-31T12:51:12Z)
• Disentangling Knowledge Representations for Large Language Model Editing [38.244171146682206]
  We propose DiKE, a novel approach that Disentangles Knowledge representations for LLM Editing.<n>DiKE consists of two key components: a Knowledge Representation Disentanglement (KRD) module that decomposes the subject representation into target-knowledgerelated and -unrelated components, and a Knowledge Edit (DKE) module that updates only the target-related component while explicitly preserving the unrelated one.<n>To rigorously evaluate fine-grained irrelevant knowledge preservation, we construct FINE-KED, a new benchmark comprising fine-grained irrelevant knowledge at different levels of relational similarity to the edited knowledge.
  arXiv  Detail & Related papers  (2025-05-24T16:24:04Z)
• Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoning [38.018263569983226]
  We provide an evaluation of ten model editing methods along four dimensions: reliability, generalization, locality, and portability.<n>We then propose a straightforward method called Selective Contextual Reasoning (SCR) for knowledge updating.
  arXiv  Detail & Related papers  (2025-03-07T08:04:25Z)
• How Do LLMs Acquire New Knowledge? A Knowledge Circuits Perspective on Continual Pre-Training [92.88889953768455]
  Large Language Models (LLMs) face a critical gap in understanding how they internalize new knowledge.<n>We identify computational subgraphs that facilitate knowledge storage and processing.
  arXiv  Detail & Related papers  (2025-02-16T16:55:43Z)
• Resolving Editing-Unlearning Conflicts: A Knowledge Codebook Framework for Large Language Model Updating [61.70705744491162]
  Large Language Models (LLMs) excel in natural language processing by encoding extensive human knowledge. Updating LLMs involves two key tasks simultaneously: unlearning to remove unwanted knowledge and editing to incorporate new information. We propose LOKA, a conflict-free framework for LLM updating based on a knowledge codebook.
  arXiv  Detail & Related papers  (2025-01-31T20:48:46Z)
• KIF: Knowledge Identification and Fusion for Language Model Continual Learning [41.28933724210434]
  We introduce a novel framework for language models, named Knowledge Identification and Fusion (KIF) KIF segregates the model into'skill units' based on parameter dependencies, allowing for more precise control. It employs a novel group-wise knowledge identification technique to ascertain the importance distribution of skill units for a new task. As a result, KIF achieves an optimal balance between retaining prior knowledge and excelling in new tasks.
  arXiv  Detail & Related papers  (2024-08-09T17:44:45Z)
• Knowledge Editing in Language Models via Adapted Direct Preference Optimization [50.616875565173274]
  Large Language Models (LLMs) can become outdated over time. Knowledge Editing aims to overcome this challenge using weight updates that do not require expensive retraining.
  arXiv  Detail & Related papers  (2024-06-14T11:02:21Z)
• Time Sensitive Knowledge Editing through Efficient Finetuning [35.79991957163508]
  Large Language Models (LLMs) have demonstrated impressive capability in different tasks and are bringing transformative changes to many domains. Keeping the knowledge in LLMs up-to-date remains a challenge once pretraining is complete. Existing locate-and-edit knowledge editing (KE) method suffers from two limitations.
  arXiv  Detail & Related papers  (2024-06-06T20:41:36Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)
• DeepEdit: Knowledge Editing as Decoding with Constraints [118.78008395850888]
  How to edit the knowledge in multi-step reasoning has become the major challenge in the knowledge editing (KE) of large language models (LLMs) We propose a new KE framework: DEEPEDIT, which enhances LLMs's ability to generate coherent reasoning chains with new knowledge through depth-first search. In addition to DEEPEDIT, we propose two new KE benchmarks: MQUAKE-2002 and MQUAKE-HARD, which provide more precise and challenging assessments of KE approaches.
  arXiv  Detail & Related papers  (2024-01-19T03:48:27Z)
• A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
  Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches. We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
  arXiv  Detail & Related papers  (2024-01-02T16:54:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.