Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing

• URL: http://arxiv.org/abs/2410.06331v2
• Date: Fri, 18 Oct 2024 17:53:46 GMT
• Title: Locate-then-edit for Multi-hop Factual Recall under Knowledge Editing
• Authors: Zhuoran Zhang, Yongxiang Li, Zijian Kan, Keyuan Cheng, Lijie Hu, Di Wang,
• Abstract summary: locate-then-edit paradigm has shown significant promise for knowledge editing. Previous methods struggle with multi-hop factual recall tasks involving newly edited knowledge. We propose IFMET, a novel locate-then-edit approach designed to edit both shallow and deep layers.
• Score: 7.9525115640025055
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The locate-then-edit paradigm has shown significant promise for knowledge editing (KE) in Large Language Models (LLMs). While previous methods perform well on single-hop fact recall tasks, they consistently struggle with multi-hop factual recall tasks involving newly edited knowledge. In this paper, leveraging tools in mechanistic interpretability, we first identify that in multi-hop tasks, LLMs tend to retrieve implicit subject knowledge from deeper MLP layers, unlike single-hop tasks, which rely on earlier layers. This distinction explains the poor performance of current methods in multi-hop queries, as they primarily focus on editing shallow layers, leaving deeper layers unchanged. To address this, we propose IFMET, a novel locate-then-edit KE approach designed to edit both shallow and deep MLP layers. IFMET employs multi-hop editing prompts and supplementary sets to locate and modify knowledge across different reasoning stages. Experimental results demonstrate that IFMET significantly improves performance on multi-hop factual recall tasks, effectively overcoming the limitations of previous locate-then-edit methods.

Related papers

• LLM-Based Multi-Hop Question Answering with Knowledge Graph Integration in Evolving Environments [35.3938477255058]
  This paper introduces Graph Memory-based Editing for Large Language Models (GMeLLo) GMeLLo merges the explicit knowledge representation of Knowledge Graphs with the linguistic flexibility of Large Language Models. Our results show that GMeLLo significantly surpasses current state-of-the-art knowledge editing methods in the multi-hop question answering benchmark, MQuAKE.
  arXiv  Detail & Related papers  (2024-08-28T16:15:45Z)
• Enhancing Multi-hop Reasoning through Knowledge Erasure in Large Language Model Editing [38.590823330865845]
  Large language models (LLMs) face challenges with internal knowledge inaccuracies and outdated information. Knowledge editing has emerged as a pivotal approach to mitigate these issues. We propose a novel knowledge editing method that incorporates a Knowledge Erasure mechanism for Large language model Editing (KELE)
  arXiv  Detail & Related papers  (2024-08-22T14:53:33Z)
• Cross-Lingual Multi-Hop Knowledge Editing -- Benchmarks, Analysis and a Simple Contrastive Learning based Approach [53.028586843468915]
  We propose the Cross-Lingual Multi-Hop Knowledge Editing paradigm, for measuring and analyzing the performance of various SoTA knowledge editing techniques in a cross-lingual setup. Specifically, we create a parallel cross-lingual benchmark, CROLIN-MQUAKE for measuring the knowledge editing capabilities. Following this, we propose a significantly improved system for cross-lingual multi-hop knowledge editing, CLEVER-CKE.
  arXiv  Detail & Related papers  (2024-07-14T17:18:16Z)
• MC-MKE: A Fine-Grained Multimodal Knowledge Editing Benchmark Emphasizing Modality Consistency [50.40318712497071]
  Multimodal large language models (MLLMs) are prone to non-factual or outdated knowledge issues. We decompose multimodal knowledge into its visual and textual components. We present MC-MKE, a fine-grained Multimodal Knowledge Editing benchmark.
  arXiv  Detail & Related papers  (2024-06-19T05:15:21Z)
• Time Sensitive Knowledge Editing through Efficient Finetuning [35.79991957163508]
  Large Language Models (LLMs) have demonstrated impressive capability in different tasks and are bringing transformative changes to many domains. Keeping the knowledge in LLMs up-to-date remains a challenge once pretraining is complete. Existing locate-and-edit knowledge editing (KE) method suffers from two limitations.
  arXiv  Detail & Related papers  (2024-06-06T20:41:36Z)
• Learning to Edit: Aligning LLMs with Knowledge Editing [101.96620267293731]
  We propose a Learning to Edit (LTE) framework, focusing on teaching large language models to apply updated knowledge into input questions. LTE features a two-phase process: (i) the Alignment Phase, which fine-tunes LLMs on a meticulously curated parallel dataset to make reliable, in-scope edits. We demonstrate LTE's superiority in knowledge editing performance, robustness in both batch and sequential editing, minimal interference on general tasks, and rapid editing speeds.
  arXiv  Detail & Related papers  (2024-02-19T07:45:17Z)
• PokeMQA: Programmable knowledge editing for Multi-hop Question Answering [46.80110170981976]
  Multi-hop question answering (MQA) is one of the challenging tasks to evaluate machine's comprehension and reasoning abilities. We propose a framework, Programmable knowledge editing for Multi-hop Question Answering (MQA) Specifically, we prompt LLMs to decompose knowledge-augmented multi-hop question, while interacting with a detached trainable scope detector to modulate LLMs behavior depending on external conflict signal.
  arXiv  Detail & Related papers  (2023-12-23T08:32:13Z)
• Editing Large Language Models: Problems, Methods, and Opportunities [51.903537096207]
  This paper embarks on a deep exploration of the problems, methods, and opportunities related to model editing for LLMs. We provide an exhaustive overview of the task definition and challenges associated with model editing, along with an in-depth empirical analysis of the most progressive methods currently at our disposal. Our objective is to provide valuable insights into the effectiveness and feasibility of each editing technique, thereby assisting the community in making informed decisions on the selection of the most appropriate method for a specific task or context.
  arXiv  Detail & Related papers  (2023-05-22T16:00:00Z)
• Rethinking Label Smoothing on Multi-hop Question Answering [87.68071401870283]
  Multi-Hop Question Answering (MHQA) is a significant area in question answering. In this work, we analyze the primary factors limiting the performance of multi-hop reasoning. We propose a novel label smoothing technique, F1 Smoothing, which incorporates uncertainty into the learning process.
  arXiv  Detail & Related papers  (2022-12-19T14:48:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.