Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing

• URL: http://arxiv.org/abs/2410.06331v3
• Date: Sat, 01 Feb 2025 12:21:59 GMT
• Title: Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing
• Authors: Zhuoran Zhang, Yongxiang Li, Zijian Kan, Keyuan Cheng, Lijie Hu, Di Wang,
• Abstract summary: locate-then-edit paradigm has shown significant promise for knowledge editing. Previous methods struggle with multi-hop factual recall tasks involving newly edited knowledge. We propose IFMET, a novel approach designed to edit both shallow and deep layers.
• Score: 7.9525115640025055
• License:
• Abstract: The locate-then-edit paradigm has shown significant promise for knowledge editing (KE) in Large Language Models (LLMs). While previous methods perform well on single-hop fact recall tasks, they consistently struggle with multi-hop factual recall tasks involving newly edited knowledge. In this paper, leveraging tools in mechanistic interpretability, we first identify that in multi-hop tasks, LLMs tend to retrieve knowledge with implicit subject information from deeper MLP layers, unlike single-hop tasks, which rely on shallow layers. This distinction explains the poor performance of current methods in multi-hop queries, as they primarily focus on editing shallow layers with single-hop edit prompts, leaving deeper layers unchanged. To address this, we propose IFMET, a novel locate-then-edit KE approach designed to edit both shallow and deep MLP layers. Beyond single-hop editing prompts, IFMET further incorporates multi-hop editing prompts to locate and modify knowledge across different stages of reasoning. Experimental results demonstrate that IFMET significantly improves performance on multi-hop factual recall tasks, overcoming the limitations of previous locate-then-edit methods

Related papers

• LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments [35.3938477255058]
  This paper introduces Graph Memory-based Editing for Large Language Models (GMeLLo) It is a straightforward and effective method that merges the explicit knowledge representation of Knowledge Graphs with the linguistic flexibility of Large Language Models. Our results show that GMeLLo significantly surpasses current state-of-the-art knowledge editing methods in the multi-hop question answering benchmark, MQuAKE.
  arXiv  Detail & Related papers  (2024-08-28T16:15:45Z)
• Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing [38.590823330865845]
  Large language models (LLMs) face challenges with internal knowledge inaccuracies and outdated information. Knowledge editing has emerged as a pivotal approach to mitigate these issues. We propose a novel knowledge editing method that incorporates a Knowledge Erasure mechanism for Large language model Editing (KELE)
  arXiv  Detail & Related papers  (2024-08-22T14:53:33Z)
• Cross-Lingual Multi-Hop Knowledge Editing [53.028586843468915]
  We propose the Cross-Lingual Multi-Hop Knowledge Editing paradigm, for measuring and analyzing the performance of various SoTA knowledge editing techniques in a cross-lingual setup. Specifically, we create a parallel cross-lingual benchmark, CROLIN-MQUAKE for measuring the knowledge editing capabilities. Following this, we propose a significantly improved system for cross-lingual multi-hop knowledge editing, CLEVER-CKE.
  arXiv  Detail & Related papers  (2024-07-14T17:18:16Z)
• MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency [50.40318712497071]
  Multimodal large language models (MLLMs) are prone to non-factual or outdated knowledge issues. We decompose multimodal knowledge into its visual and textual components. We present MC-MKE, a fine-grained Multimodal Knowledge Editing benchmark.
  arXiv  Detail & Related papers  (2024-06-19T05:15:21Z)
• Time Sensitive Knowledge Editing through Efficient Finetuning [35.79991957163508]
  Large Language Models (LLMs) have demonstrated impressive capability in different tasks and are bringing transformative changes to many domains. Keeping the knowledge in LLMs up-to-date remains a challenge once pretraining is complete. Existing locate-and-edit knowledge editing (KE) method suffers from two limitations.
  arXiv  Detail & Related papers  (2024-06-06T20:41:36Z)
• Learning to Edit: Aligning LLMs with Knowledge Editing [101.96620267293731]
  We propose a Learning to Edit (LTE) framework, focusing on teaching large language models to apply updated knowledge into input questions. LTE features a two-phase process: (i) the Alignment Phase, which fine-tunes LLMs on a meticulously curated parallel dataset to make reliable, in-scope edits. We demonstrate LTE's superiority in knowledge editing performance, robustness in both batch and sequential editing, minimal interference on general tasks, and rapid editing speeds.
  arXiv  Detail & Related papers  (2024-02-19T07:45:17Z)
• On the Robustness of Editing Large Language Models [57.477943944826904]
  Large language models (LLMs) have played a pivotal role in building communicative AI, yet they encounter the challenge of efficient updates. This work seeks to understand the strengths and limitations of editing methods, facilitating practical applications of communicative AI.
  arXiv  Detail & Related papers  (2024-02-08T17:06:45Z)
• PokeMQA: Programmable knowledge editing for Multi-hop Question Answering [46.80110170981976]
  Multi-hop question answering (MQA) is one of the challenging tasks to evaluate machine's comprehension and reasoning abilities. We propose a framework, Programmable knowledge editing for Multi-hop Question Answering (MQA) Specifically, we prompt LLMs to decompose knowledge-augmented multi-hop question, while interacting with a detached trainable scope detector to modulate LLMs behavior depending on external conflict signal.
  arXiv  Detail & Related papers  (2023-12-23T08:32:13Z)
• MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions [75.21713251369225]
  We present a benchmark, MQuAKE, comprising multi-hop questions that assess whether edited models correctly answer questions. We propose a memory-based approach, MeLLo, which stores all edited facts externally while prompting the language model iteratively to generate answers consistent with the edited facts.
  arXiv  Detail & Related papers  (2023-05-24T06:48:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.