$μ$KE: Matryoshka Unstructured Knowledge Editing of Large Language Models

• URL: http://arxiv.org/abs/2504.01196v1
• Date: Tue, 01 Apr 2025 21:24:44 GMT
• Title: $μ$KE: Matryoshka Unstructured Knowledge Editing of Large Language Models
• Authors: Zian Su, Ziyang Huang, Kaiyuan Zhang, Xiangyu Zhang,
• Abstract summary: Matryoshka Unstructured Knowledge Editing preserves dependencies between memory updates and output tokens.<n>$mu$KE improves edit efficacy by up to 12.33% over state-of-the-art methods.
• Score: 8.472795721252856
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have emerged as powerful knowledge bases yet are limited by static training data, leading to issues such as hallucinations and safety risks. Editing a model's internal knowledge through the locate-and-edit paradigm has proven a cost-effective alternative to retraining, though current unstructured approaches, especially window-based autoregressive methods, often disrupt the causal dependency between early memory updates and later output tokens. In this work, we first theoretically analyze these limitations and then introduce Matryoshka Unstructured Knowledge Editing ($\mu$KE), a novel memory update mechanism that preserves such dependencies via a Matryoshka-style objective and adaptive loss coefficients. Empirical evaluations on two models across four benchmarks demonstrate that $\mu$KE improves edit efficacy by up to 12.33% over state-of-the-art methods, and remain robust when applied to diverse formatted edits, underscoring its potential for effective unstructured knowledge editing in LLMs.

Related papers

• Model Hemorrhage and the Robustness Limits of Large Language Models [119.46442117681147]
  Large language models (LLMs) demonstrate strong performance across natural language processing tasks, yet undergo significant performance degradation when modified for deployment. We define this phenomenon as model hemorrhage - performance decline caused by parameter alterations and architectural changes.
  arXiv  Detail & Related papers  (2025-03-31T10:16:03Z)
• CaKE: Circuit-aware Editing Enables Generalizable Knowledge Learners [88.35958039968081]
  CaKE (Circuit-aware Knowledge Editing) is a novel method that enables more effective integration of updated knowledge in large language models. Results show that CaKE enables more accurate and consistent use of updated knowledge across related reasoning tasks.
  arXiv  Detail & Related papers  (2025-03-20T17:14:34Z)
• Knowledge Updating? No More Model Editing! Just Selective Contextual Reasoning [38.018263569983226]
  We provide an evaluation of ten model editing methods along four dimensions: reliability, generalization, locality, and portability.<n>We then propose a straightforward method called Selective Contextual Reasoning (SCR) for knowledge updating.
  arXiv  Detail & Related papers  (2025-03-07T08:04:25Z)
• Keys to Robust Edits: from Theoretical Insights to Practical Advances [20.10464264597003]
  Large language models (LLMs) have revolutionized knowledge storage and retrieval, but face challenges with conflicting and outdated information. This work investigates the cause of failures in locate-and-edit methods, offering theoretical insights into their key-value modeling. Empirical analysis reveals that keys used by current methods fail to meet robustness and specificity requirements.
  arXiv  Detail & Related papers  (2024-10-12T02:54:12Z)
• Outdated Issue Aware Decoding for Reasoning Questions on Edited Knowledge [93.54427119091174]
  We propose outDated ISsue aware deCOding to enhance the performance of edited models on reasoning questions. We capture the difference in the probability distribution between the original and edited models. We amplify the difference of the token prediction in the edited model to alleviate the outdated issue.
  arXiv  Detail & Related papers  (2024-06-05T03:00:15Z)
• Everything is Editable: Extend Knowledge Editing to Unstructured Data in Large Language Models [65.10456412127405]
  We propose a novel Unstructured Knowledge Editing method, namely UnKE.<n>In the layer dimension, we propose non-local block key-value storage to replace local layer key-value storage.<n>In the token dimension, we replace "term-driven optimization" with "cause-driven optimization", which edits the last token directly while preserving context.
  arXiv  Detail & Related papers  (2024-05-24T08:42:40Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)
• SWEA: Updating Factual Knowledge in Large Language Models via Subject Word Embedding Altering [17.20346072074533]
  Recent model editing is a promising technique for efficiently updating a small amount of knowledge of large language models.<n>We propose a detachable and expandable Subject Word Embedding Altering (SWEA) framework, which finds the editing embeddings through token-level matching.<n>We demonstrate the overall state-of-the-art (SOTA) performance of SWEA$oplus$OS on the CounterFact and zsRE datasets.
  arXiv  Detail & Related papers  (2024-01-31T13:08:45Z)
• A Comprehensive Study of Knowledge Editing for Large Language Models [82.65729336401027]
  Large Language Models (LLMs) have shown extraordinary capabilities in understanding and generating text that closely mirrors human communication. This paper defines the knowledge editing problem and provides a comprehensive review of cutting-edge approaches. We introduce a new benchmark, KnowEdit, for a comprehensive empirical evaluation of representative knowledge editing approaches.
  arXiv  Detail & Related papers  (2024-01-02T16:54:58Z)
• A generalized framework for active learning reliability: survey and benchmark [0.0]
  We propose a modular framework to build on-the-fly efficient active learning strategies. We devise 39 strategies for the solution of 20 reliability benchmark problems.
  arXiv  Detail & Related papers  (2021-06-03T09:33:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.